<!doctype html>
<html style='font-size:20px !important'>
<head>
<meta charset='UTF-8'><meta name='viewport' content='width=device-width initial-scale=1'>
<title>6.chapter_six</title><style type='text/css'>html {overflow-x: initial !important;}:root { --bg-color:#ffffff; --text-color:#333333; --select-text-bg-color:#B5D6FC; --select-text-font-color:auto; --monospace:"Lucida Console",Consolas,"Courier",monospace; }
html { font-size: 14px; background-color: var(--bg-color); color: var(--text-color); font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; -webkit-font-smoothing: antialiased; }
body { margin: 0px; padding: 0px; height: auto; bottom: 0px; top: 0px; left: 0px; right: 0px; font-size: 1rem; line-height: 1.42857; overflow-x: hidden; background: inherit; tab-size: 4; }
iframe { margin: auto; }
a.url { word-break: break-all; }
a:active, a:hover { outline: 0px; }
.in-text-selection, ::selection { text-shadow: none; background: var(--select-text-bg-color); color: var(--select-text-font-color); }
#write { margin: 0px auto; height: auto; width: inherit; word-break: normal; overflow-wrap: break-word; position: relative; white-space: normal; overflow-x: visible; padding-top: 40px; }
#write.first-line-indent p { text-indent: 2em; }
#write.first-line-indent li p, #write.first-line-indent p * { text-indent: 0px; }
#write.first-line-indent li { margin-left: 2em; }
.for-image #write { padding-left: 8px; padding-right: 8px; }
body.typora-export { padding-left: 30px; padding-right: 30px; }
.typora-export .footnote-line, .typora-export li, .typora-export p { white-space: pre-wrap; }
.typora-export .task-list-item input { pointer-events: none; }
@media screen and (max-width: 500px) {
  body.typora-export { padding-left: 0px; padding-right: 0px; }
  #write { padding-left: 20px; padding-right: 20px; }
  .CodeMirror-sizer { margin-left: 0px !important; }
  .CodeMirror-gutters { display: none !important; }
}
#write li > figure:last-child { margin-bottom: 0.5rem; }
#write ol, #write ul { position: relative; }
img { max-width: 100%; vertical-align: middle; image-orientation: from-image; }
button, input, select, textarea { color: inherit; font: inherit; }
input[type="checkbox"], input[type="radio"] { line-height: normal; padding: 0px; }
*, ::after, ::before { box-sizing: border-box; }
#write h1, #write h2, #write h3, #write h4, #write h5, #write h6, #write p, #write pre { width: inherit; }
#write h1, #write h2, #write h3, #write h4, #write h5, #write h6, #write p { position: relative; }
p { line-height: inherit; }
h1, h2, h3, h4, h5, h6 { break-after: avoid-page; break-inside: avoid; orphans: 4; }
p { orphans: 4; }
h1 { font-size: 2rem; }
h2 { font-size: 1.8rem; }
h3 { font-size: 1.6rem; }
h4 { font-size: 1.4rem; }
h5 { font-size: 1.2rem; }
h6 { font-size: 1rem; }
.md-math-block, .md-rawblock, h1, h2, h3, h4, h5, h6, p { margin-top: 1rem; margin-bottom: 1rem; }
.hidden { display: none; }
.md-blockmeta { color: rgb(204, 204, 204); font-weight: 700; font-style: italic; }
a { cursor: pointer; }
sup.md-footnote { padding: 2px 4px; background-color: rgba(238, 238, 238, 0.7); color: rgb(85, 85, 85); border-radius: 4px; cursor: pointer; }
sup.md-footnote a, sup.md-footnote a:hover { color: inherit; text-transform: inherit; text-decoration: inherit; }
#write input[type="checkbox"] { cursor: pointer; width: inherit; height: inherit; }
figure { overflow-x: auto; margin: 1.2em 0px; max-width: calc(100% + 16px); padding: 0px; }
figure > table { margin: 0px; }
tr { break-inside: avoid; break-after: auto; }
thead { display: table-header-group; }
table { border-collapse: collapse; border-spacing: 0px; width: 100%; overflow: auto; break-inside: auto; text-align: left; }
table.md-table td { min-width: 32px; }
.CodeMirror-gutters { border-right: 0px; background-color: inherit; }
.CodeMirror-linenumber { user-select: none; }
.CodeMirror { text-align: left; }
.CodeMirror-placeholder { opacity: 0.3; }
.CodeMirror pre { padding: 0px 4px; }
.CodeMirror-lines { padding: 0px; }
div.hr:focus { cursor: none; }
#write pre { white-space: pre-wrap; }
#write.fences-no-line-wrapping pre { white-space: pre; }
#write pre.ty-contain-cm { white-space: normal; }
.CodeMirror-gutters { margin-right: 4px; }
.md-fences { font-size: 0.9rem; display: block; break-inside: avoid; text-align: left; overflow: visible; white-space: pre; background: inherit; position: relative !important; }
.md-diagram-panel { width: 100%; margin-top: 10px; text-align: center; padding-top: 0px; padding-bottom: 8px; overflow-x: auto; }
#write .md-fences.mock-cm { white-space: pre-wrap; }
.md-fences.md-fences-with-lineno { padding-left: 0px; }
#write.fences-no-line-wrapping .md-fences.mock-cm { white-space: pre; overflow-x: auto; }
.md-fences.mock-cm.md-fences-with-lineno { padding-left: 8px; }
.CodeMirror-line, twitterwidget { break-inside: avoid; }
.footnotes { opacity: 0.8; font-size: 0.9rem; margin-top: 1em; margin-bottom: 1em; }
.footnotes + .footnotes { margin-top: 0px; }
.md-reset { margin: 0px; padding: 0px; border: 0px; outline: 0px; vertical-align: top; background: 0px 0px; text-decoration: none; text-shadow: none; float: none; position: static; width: auto; height: auto; white-space: nowrap; cursor: inherit; -webkit-tap-highlight-color: transparent; line-height: normal; font-weight: 400; text-align: left; box-sizing: content-box; direction: ltr; }
li div { padding-top: 0px; }
blockquote { margin: 1rem 0px; }
li .mathjax-block, li p { margin: 0.5rem 0px; }
li { margin: 0px; position: relative; }
blockquote > :last-child { margin-bottom: 0px; }
blockquote > :first-child, li > :first-child { margin-top: 0px; }
.footnotes-area { color: rgb(136, 136, 136); margin-top: 0.714rem; padding-bottom: 0.143rem; white-space: normal; }
#write .footnote-line { white-space: pre-wrap; }
@media print {
  body, html { border: 1px solid transparent; height: 99%; break-after: avoid; break-before: avoid; font-variant-ligatures: no-common-ligatures; }
  #write { margin-top: 0px; padding-top: 0px; border-color: transparent !important; }
  .typora-export * { -webkit-print-color-adjust: exact; }
  html.blink-to-pdf { font-size: 13px; }
  .typora-export #write { break-after: avoid; }
  .typora-export #write::after { height: 0px; }
  .is-mac table { break-inside: avoid; }
}
.footnote-line { margin-top: 0.714em; font-size: 0.7em; }
a img, img a { cursor: pointer; }
pre.md-meta-block { font-size: 0.8rem; min-height: 0.8rem; white-space: pre-wrap; background: rgb(204, 204, 204); display: block; overflow-x: hidden; }
p > .md-image:only-child:not(.md-img-error) img, p > img:only-child { display: block; margin: auto; }
#write.first-line-indent p > .md-image:only-child:not(.md-img-error) img { left: -2em; position: relative; }
p > .md-image:only-child { display: inline-block; width: 100%; }
#write .MathJax_Display { margin: 0.8em 0px 0px; }
.md-math-block { width: 100%; }
.md-math-block:not(:empty)::after { display: none; }
[contenteditable="true"]:active, [contenteditable="true"]:focus, [contenteditable="false"]:active, [contenteditable="false"]:focus { outline: 0px; box-shadow: none; }
.md-task-list-item { position: relative; list-style-type: none; }
.task-list-item.md-task-list-item { padding-left: 0px; }
.md-task-list-item > input { position: absolute; top: 0px; left: 0px; margin-left: -1.2em; margin-top: calc(1em - 10px); border: none; }
.math { font-size: 1rem; }
.md-toc { min-height: 3.58rem; position: relative; font-size: 0.9rem; border-radius: 10px; }
.md-toc-content { position: relative; margin-left: 0px; }
.md-toc-content::after, .md-toc::after { display: none; }
.md-toc-item { display: block; color: rgb(65, 131, 196); }
.md-toc-item a { text-decoration: none; }
.md-toc-inner:hover { text-decoration: underline; }
.md-toc-inner { display: inline-block; cursor: pointer; }
.md-toc-h1 .md-toc-inner { margin-left: 0px; font-weight: 700; }
.md-toc-h2 .md-toc-inner { margin-left: 2em; }
.md-toc-h3 .md-toc-inner { margin-left: 4em; }
.md-toc-h4 .md-toc-inner { margin-left: 6em; }
.md-toc-h5 .md-toc-inner { margin-left: 8em; }
.md-toc-h6 .md-toc-inner { margin-left: 10em; }
@media screen and (max-width: 48em) {
  .md-toc-h3 .md-toc-inner { margin-left: 3.5em; }
  .md-toc-h4 .md-toc-inner { margin-left: 5em; }
  .md-toc-h5 .md-toc-inner { margin-left: 6.5em; }
  .md-toc-h6 .md-toc-inner { margin-left: 8em; }
}
a.md-toc-inner { font-size: inherit; font-style: inherit; font-weight: inherit; line-height: inherit; }
.footnote-line a:not(.reversefootnote) { color: inherit; }
.md-attr { display: none; }
.md-fn-count::after { content: "."; }
code, pre, samp, tt { font-family: var(--monospace); }
kbd { margin: 0px 0.1em; padding: 0.1em 0.6em; font-size: 0.8em; color: rgb(36, 39, 41); background: rgb(255, 255, 255); border: 1px solid rgb(173, 179, 185); border-radius: 3px; box-shadow: rgba(12, 13, 14, 0.2) 0px 1px 0px, rgb(255, 255, 255) 0px 0px 0px 2px inset; white-space: nowrap; vertical-align: middle; }
.md-comment { color: rgb(162, 127, 3); opacity: 0.8; font-family: var(--monospace); }
code { text-align: left; vertical-align: initial; }
a.md-print-anchor { white-space: pre !important; border-width: initial !important; border-style: none !important; border-color: initial !important; display: inline-block !important; position: absolute !important; width: 1px !important; right: 0px !important; outline: 0px !important; background: 0px 0px !important; text-decoration: initial !important; text-shadow: initial !important; }
.md-inline-math .MathJax_SVG .noError { display: none !important; }
.html-for-mac .inline-math-svg .MathJax_SVG { vertical-align: 0.2px; }
.md-math-block .MathJax_SVG_Display { text-align: center; margin: 0px; position: relative; text-indent: 0px; max-width: none; max-height: none; min-height: 0px; min-width: 100%; width: auto; overflow-y: hidden; display: block !important; }
.MathJax_SVG_Display, .md-inline-math .MathJax_SVG_Display { width: auto; margin: inherit; display: inline-block !important; }
.MathJax_SVG .MJX-monospace { font-family: var(--monospace); }
.MathJax_SVG .MJX-sans-serif { font-family: sans-serif; }
.MathJax_SVG { display: inline; font-style: normal; font-weight: 400; line-height: normal; zoom: 90%; text-indent: 0px; text-align: left; text-transform: none; letter-spacing: normal; word-spacing: normal; overflow-wrap: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0px; min-height: 0px; border: 0px; padding: 0px; margin: 0px; }
.MathJax_SVG * { transition: none 0s ease 0s; }
.MathJax_SVG_Display svg { vertical-align: middle !important; margin-bottom: 0px !important; margin-top: 0px !important; }
.os-windows.monocolor-emoji .md-emoji { font-family: "Segoe UI Symbol", sans-serif; }
.md-diagram-panel > svg { max-width: 100%; }
[lang="flow"] svg, [lang="mermaid"] svg { max-width: 100%; height: auto; }
[lang="mermaid"] .node text { font-size: 1rem; }
table tr th { border-bottom: 0px; }
video { max-width: 100%; display: block; margin: 0px auto; }
iframe { max-width: 100%; width: 100%; border: none; }
.highlight td, .highlight tr { border: 0px; }
svg[id^="mermaidChart"] { line-height: 1em; }
mark { background: rgb(255, 255, 0); color: rgb(0, 0, 0); }
.md-html-inline .md-plain, .md-html-inline strong, mark .md-inline-math, mark strong { color: inherit; }
mark .md-meta { color: rgb(0, 0, 0); opacity: 0.3 !important; }
@media print {
  .typora-export h1, .typora-export h2, .typora-export h3, .typora-export h4, .typora-export h5, .typora-export h6 { break-inside: avoid; }
}


/* Flowchart variables */
/* Sequence Diagram variables */
/* Gantt chart variables */
/* state colors */
.label {
  
  color: #333; }

.label text {
  fill: #333; }

.node rect,
.node circle,
.node ellipse,
.node polygon {
  fill: #BDD5EA;
  stroke: #9370DB;
  stroke-width: 1px; }

.node .label {
  text-align: center; }

.node.clickable {
  cursor: pointer; }

.arrowheadPath {
  fill: lightgrey; }

.edgePath .path {
  stroke: lightgrey;
  stroke-width: 1.5px; }

.edgeLabel {
  background-color: #e8e8e8;
  text-align: center; }

.cluster rect {
  fill: #6D6D65;
  stroke: rgba(255, 255, 255, 0.25);
  stroke-width: 1px; }

.cluster text {
  fill: #F9FFFE; }

div.mermaidTooltip {
  position: absolute;
  text-align: center;
  max-width: 200px;
  padding: 2px;
  
  font-size: 12px;
  background: #6D6D65;
  border: 1px solid rgba(255, 255, 255, 0.25);
  border-radius: 2px;
  pointer-events: none;
  z-index: 100; }

.actor {
  stroke: #81B1DB;
  fill: #BDD5EA; }

text.actor {
  fill: black;
  stroke: none; }

.actor-line {
  stroke: lightgrey; }

.messageLine0 {
  stroke-width: 1.5;
  stroke-dasharray: '2 2';
  stroke: lightgrey; }

.messageLine1 {
  stroke-width: 1.5;
  stroke-dasharray: '2 2';
  stroke: lightgrey; }

#arrowhead {
  fill: lightgrey; }

.sequenceNumber {
  fill: white; }

#sequencenumber {
  fill: lightgrey; }

#crosshead path {
  fill: lightgrey !important;
  stroke: lightgrey !important; }

.messageText {
  fill: lightgrey;
  stroke: none; }

.labelBox {
  stroke: #81B1DB;
  fill: #BDD5EA; }

.labelText {
  fill: #323D47;
  stroke: none; }

.loopText {
  fill: lightgrey;
  stroke: none; }

.loopLine {
  stroke-width: 2;
  stroke-dasharray: '2 2';
  stroke: #81B1DB; }

.note {
  stroke: rgba(255, 255, 255, 0.25);
  fill: #fff5ad; }

.noteText {
  fill: black;
  stroke: none;
  
  font-size: 14px; }

.activation0 {
  fill: #f4f4f4;
  stroke: #666; }

.activation1 {
  fill: #f4f4f4;
  stroke: #666; }

.activation2 {
  fill: #f4f4f4;
  stroke: #666; }

/** Section styling */
.section {
  stroke: none;
  opacity: 0.2; }

.section0 {
  fill: rgba(255, 255, 255, 0.3); }

.section2 {
  fill: #EAE8B9; }

.section1,
.section3 {
  fill: white;
  opacity: 0.2; }

.sectionTitle0 {
  fill: #F9FFFE; }

.sectionTitle1 {
  fill: #F9FFFE; }

.sectionTitle2 {
  fill: #F9FFFE; }

.sectionTitle3 {
  fill: #F9FFFE; }

.sectionTitle {
  text-anchor: start;
  font-size: 11px;
  text-height: 14px;
   }

/* Grid and axis */
.grid .tick {
  stroke: lightgrey;
  opacity: 0.3;
  shape-rendering: crispEdges; }

.grid path {
  stroke-width: 0; }

/* Today line */
.today {
  fill: none;
  stroke: #DB5757;
  stroke-width: 2px; }

/* Task styling */
/* Default task */
.task {
  stroke-width: 2; }

.taskText {
  text-anchor: middle;
   }

.taskText:not([font-size]) {
  font-size: 11px; }

.taskTextOutsideRight {
  fill: #323D47;
  text-anchor: start;
  font-size: 11px;
   }

.taskTextOutsideLeft {
  fill: #323D47;
  text-anchor: end;
  font-size: 11px; }

/* Special case clickable */
.task.clickable {
  cursor: pointer; }

.taskText.clickable {
  cursor: pointer;
  fill: #003163 !important;
  font-weight: bold; }

.taskTextOutsideLeft.clickable {
  cursor: pointer;
  fill: #003163 !important;
  font-weight: bold; }

.taskTextOutsideRight.clickable {
  cursor: pointer;
  fill: #003163 !important;
  font-weight: bold; }

/* Specific task settings for the sections*/
.taskText0,
.taskText1,
.taskText2,
.taskText3 {
  fill: #323D47; }

.task0,
.task1,
.task2,
.task3 {
  fill: #BDD5EA;
  stroke: rgba(255, 255, 255, 0.5); }

.taskTextOutside0,
.taskTextOutside2 {
  fill: lightgrey; }

.taskTextOutside1,
.taskTextOutside3 {
  fill: lightgrey; }

/* Active task */
.active0,
.active1,
.active2,
.active3 {
  fill: #81B1DB;
  stroke: rgba(255, 255, 255, 0.5); }

.activeText0,
.activeText1,
.activeText2,
.activeText3 {
  fill: #323D47 !important; }

/* Completed task */
.done0,
.done1,
.done2,
.done3 {
  stroke: grey;
  fill: lightgrey;
  stroke-width: 2; }

.doneText0,
.doneText1,
.doneText2,
.doneText3 {
  fill: #323D47 !important; }

/* Tasks on the critical line */
.crit0,
.crit1,
.crit2,
.crit3 {
  stroke: #E83737;
  fill: #E83737;
  stroke-width: 2; }

.activeCrit0,
.activeCrit1,
.activeCrit2,
.activeCrit3 {
  stroke: #E83737;
  fill: #81B1DB;
  stroke-width: 2; }

.doneCrit0,
.doneCrit1,
.doneCrit2,
.doneCrit3 {
  stroke: #E83737;
  fill: lightgrey;
  stroke-width: 2;
  cursor: pointer;
  shape-rendering: crispEdges; }

.milestone {
  transform: rotate(45deg) scale(0.8, 0.8); }

.milestoneText {
  font-style: italic; }

.doneCritText0,
.doneCritText1,
.doneCritText2,
.doneCritText3 {
  fill: #323D47 !important; }

.activeCritText0,
.activeCritText1,
.activeCritText2,
.activeCritText3 {
  fill: #323D47 !important; }

.titleText {
  text-anchor: middle;
  font-size: 18px;
  fill: #323D47;
   }

g.classGroup text {
  fill: #9370DB;
  stroke: none;
  
  font-size: 10px; }
  g.classGroup text .title {
    font-weight: bolder; }

g.classGroup rect {
  fill: #BDD5EA;
  stroke: #9370DB; }

g.classGroup line {
  stroke: #9370DB;
  stroke-width: 1; }

.classLabel .box {
  stroke: none;
  stroke-width: 0;
  fill: #BDD5EA;
  opacity: 0.5; }

.classLabel .label {
  fill: #9370DB;
  font-size: 10px; }

.relation {
  stroke: #9370DB;
  stroke-width: 1;
  fill: none; }

#compositionStart {
  fill: #9370DB;
  stroke: #9370DB;
  stroke-width: 1; }

#compositionEnd {
  fill: #9370DB;
  stroke: #9370DB;
  stroke-width: 1; }

#aggregationStart {
  fill: #BDD5EA;
  stroke: #9370DB;
  stroke-width: 1; }

#aggregationEnd {
  fill: #BDD5EA;
  stroke: #9370DB;
  stroke-width: 1; }

#dependencyStart {
  fill: #9370DB;
  stroke: #9370DB;
  stroke-width: 1; }

#dependencyEnd {
  fill: #9370DB;
  stroke: #9370DB;
  stroke-width: 1; }

#extensionStart {
  fill: #9370DB;
  stroke: #9370DB;
  stroke-width: 1; }

#extensionEnd {
  fill: #9370DB;
  stroke: #9370DB;
  stroke-width: 1; }

.commit-id,
.commit-msg,
.branch-label {
  fill: lightgrey;
  color: lightgrey;
   }

.pieTitleText {
  text-anchor: middle;
  font-size: 25px;
  fill: #eee;
}

g.stateGroup text {
  stroke: none;
  font-size: 10px;
}

g.stateGroup circle {
  fill: white !important;
  stroke: white !important;
}

g.stateGroup .state-title {
  font-weight: bolder;
  fill: black; }

g.stateGroup rect {
  fill: #ececff;
  stroke: #9370DB; }

g.stateGroup line {
  stroke: #9370DB;
  stroke-width: 1; }

.transition {
  stroke: #9370DB;
  stroke-width: 1;
  fill: none; }

.stateGroup .composit {
  fill: #555;
  border-bottom: 1px; }

.state-note {
  stroke: rgba(255, 255, 255, 0.25);
  fill: #fff5ad; }
  .state-note text {
    fill: black;
    stroke: none;
    font-size: 10px; }

.stateLabel .box {
  stroke: none;
  stroke-width: 0;
  fill: #BDD5EA;
  opacity: 0.5; }

.stateLabel text {
  fill: black;
  font-size: 10px;
  font-weight: bold;
}

.cluster-label {
  color:black;
}

.statediagram-cluster rect {
  fill: #BDD5EA;
  stroke: #9370DB; 
  stroke-width: 1px;
}
.statediagram-cluster rect.outer {
  rx: 5px;
  ry: 5px;
}
.statediagram-state .divider {
  stroke: #9370DB; 
}

.statediagram-state .title-state {
  rx: 5px;
  ry: 5px;
}
.statediagram-cluster.statediagram-cluster .inner {
  fill: white;
}
.statediagram-cluster.statediagram-cluster-alt .inner {
  fill: #e0e0e0;
}

.statediagram-cluster .inner {
  rx:0;
  ry:0;
}

.statediagram-state rect.basic {
  rx: 5px;
  ry: 5px;
}
.statediagram-state rect.divider {
  stroke-dasharray: 10,10;
  fill: #efefef;
}

.note-edge {
  stroke-dasharray: 5;
}

.statediagram-note rect {
  stroke: var(--cluster-border);
  fill: #fff5ad;
  stroke-width: 1px;
  rx: 0;
  ry: 0;
}

.node circle.state-start {
  fill: black;
  stroke: black;
}
.node circle.state-end {
  fill: black;
  stroke: white;
  stroke-width: 1.5
}
#statediagram-barbEnd {
  fill: #9370DB; 
}

/* CSS Document */

/** code highlight */

.cm-s-inner .cm-variable,
.cm-s-inner .cm-operator,
.cm-s-inner .cm-property {
    color: #b8bfc6;
}

.cm-s-inner .cm-keyword {
    color: #C88FD0;
}

.cm-s-inner .cm-tag {
    color: #7DF46A;
}

.cm-s-inner .cm-attribute {
    color: #7575E4;
}

.CodeMirror div.CodeMirror-cursor {
    border-left: 1px solid #b8bfc6;
    z-index: 3;
}

.cm-s-inner .cm-string {
    color: #D26B6B;
}

.cm-s-inner .cm-comment,
.cm-s-inner.cm-comment {
    color: #DA924A;
}

.cm-s-inner .cm-header,
.cm-s-inner .cm-def,
.cm-s-inner.cm-header,
.cm-s-inner.cm-def {
    color: #8d8df0;
}

.cm-s-inner .cm-quote,
.cm-s-inner.cm-quote {
    color: #57ac57;
}

.cm-s-inner .cm-hr {
    color: #d8d5d5;
}

.cm-s-inner .cm-link {
    color: #d3d3ef;
}

.cm-s-inner .cm-negative {
    color: #d95050;
}

.cm-s-inner .cm-positive {
    color: #50e650;
}

.cm-s-inner .cm-string-2 {
    color: #f50;
}

.cm-s-inner .cm-meta,
.cm-s-inner .cm-qualifier {
    color: #b7b3b3;
}

.cm-s-inner .cm-builtin {
    color: #f3b3f8;
}

.cm-s-inner .cm-bracket {
    color: #997;
}

.cm-s-inner .cm-atom,
.cm-s-inner.cm-atom {
    color: #84B6CB;
}

.cm-s-inner .cm-number {
    color: #64AB8F;
}

.cm-s-inner .cm-variable {
    color: #b8bfc6;
}

.cm-s-inner .cm-variable-2 {
    color: #9FBAD5;
}

.cm-s-inner .cm-variable-3 {
    color: #1cc685;
}

.CodeMirror-selectedtext,
.CodeMirror-selected {
    background: #4a89dc;
    color: #fff !important;
    text-shadow: none;
}

.CodeMirror-gutters {
    border-right: none;
}

/* CSS Document */

/** markdown source **/
.cm-s-typora-default .cm-header, 
.cm-s-typora-default .cm-property
{
    color: #cebcca;
}

.CodeMirror.cm-s-typora-default div.CodeMirror-cursor{
    border-left: 3px solid #b8bfc6;
}

.cm-s-typora-default .cm-comment {
    color: #9FB1FF;
}

.cm-s-typora-default .cm-string {
    color: #A7A7D9
}

.cm-s-typora-default .cm-atom, .cm-s-typora-default .cm-number {
    color: #848695;
    font-style: italic;
}

.cm-s-typora-default .cm-link {
    color: #95B94B;
}

.cm-s-typora-default .CodeMirror-activeline-background {
    background: rgba(51, 51, 51, 0.72);
}

.cm-s-typora-default .cm-comment, .cm-s-typora-default .cm-code {
	color: #8aa1e1;
}@import "";
@import "";
@import "";

:root {
    --bg-color:  #363B40;
    --side-bar-bg-color: #2E3033;
    --text-color: #b8bfc6;

    --select-text-bg-color:#4a89dc;

    --item-hover-bg-color: #0a0d16;
    --control-text-color: #b7b7b7;
    --control-text-hover-color: #eee;
    --window-border: 1px solid #555;

    --active-file-bg-color: rgb(34, 34, 34);
    --active-file-border-color: #8d8df0;

    --primary-color: #a3d5fe;

    --active-file-text-color: white;
    --item-hover-bg-color: #70717d;
    --item-hover-text-color: white;
    --primary-color: #6dc1e7;

    --rawblock-edit-panel-bd: #333;

    --search-select-bg-color: #428bca;
}

html {
    font-size: 16px;
}

html,
body {
    -webkit-text-size-adjust: 100%;
    -ms-text-size-adjust: 100%;
    background: #363B40;
    background: var(--bg-color);
    fill: currentColor;
    line-height: 1.625rem;
}

#write {
    max-width: 1080px;
}


@media only screen and (min-width: 1400px) {
	#write {
		max-width: 1024px;
	}
}

@media only screen and (min-width: 1800px) {
	#write {
		max-width: 1200px;
	}
}

html,
body,
button,
input,
select,
textarea,
div.code-tooltip-content {
    color: #b8bfc6;
    border-color: transparent;
}

div.code-tooltip,
.md-hover-tip .md-arrow:after {
    background: #333;
}

.popover.bottom > .arrow:after {
    border-bottom-color: #333;
}

html,
body,
button,
input,
select,
textarea {
    font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;
}

hr {
    height: 2px;
    border: 0;
    margin: 24px 0 !important;
}

h1,
h2,
h3,
h4,
h5,
h6 {
    font-family: "Lucida Grande", "Corbel", sans-serif;
    font-weight: normal;
    clear: both;
    -ms-word-wrap: break-word;
    word-wrap: break-word;
    margin: 0;
    padding: 0;
    color: #DEDEDE
}

h1 {
    font-size: 2.5rem;
    /* 36px */
    line-height: 2.75rem;
    /* 40px */
    margin-bottom: 1.5rem;
    /* 24px */
    letter-spacing: -1.5px;
}

h2 {
    font-size: 1.63rem;
    /* 24px */
    line-height: 1.875rem;
    /* 30px */
    margin-bottom: 1.5rem;
    /* 24px */
    letter-spacing: -1px;
    font-weight: bold;
}

h3 {
    font-size: 1.17rem;
    /* 18px */
    line-height: 1.5rem;
    /* 24px */
    margin-bottom: 1.5rem;
    /* 24px */
    letter-spacing: -1px;
    font-weight: bold;
}

h4 {
    font-size: 1.12rem;
    /* 16px */
    line-height: 1.375rem;
    /* 22px */
    margin-bottom: 1.5rem;
    /* 24px */
    color: white;
}

h5 {
    font-size: 0.97rem;
    /* 16px */
    line-height: 1.25rem;
    /* 22px */
    margin-bottom: 1.5rem;
    /* 24px */
    font-weight: bold;
}

h6 {
    font-size: 0.93rem;
    /* 16px */
    line-height: 1rem;
    /* 16px */
    margin-bottom: 0.75rem;
    color: white;
}

@media (min-width: 980px) {
    h3.md-focus:before,
    h4.md-focus:before,
    h5.md-focus:before,
    h6.md-focus:before {
        color: #ddd;
        border: 1px solid #ddd;
        border-radius: 3px;
        position: absolute;
        left: -1.642857143rem;
        top: .357142857rem;
        float: left;
        font-size: 9px;
        padding-left: 2px;
        padding-right: 2px;
        vertical-align: bottom;
        font-weight: normal;
        line-height: normal;
    }

    h3.md-focus:before {
        content: 'h3';
    }

    h4.md-focus:before {
        content: 'h4';
    }

    h5.md-focus:before {
        content: 'h5';
        top: 0px;
    }

    h6.md-focus:before {
        content: 'h6';
        top: 0px;
    }
}

a {
    text-decoration: none;
    outline: 0;
}

a:hover {
    outline: 0;
}

a:focus {
    outline: thin dotted;
}

sup.md-footnote {
    background-color: #555;
    color: #ddd;
}

p {
    -ms-word-wrap: break-word;
    word-wrap: break-word;
}

p,
ul,
dd,
ol,
hr,
address,
pre,
table,
iframe,
.wp-caption,
.wp-audio-shortcode,
.wp-video-shortcode {
    margin-top: 0;
    margin-bottom: 1.5rem;
    /* 24px */
}

li > blockquote {
	margin-bottom: 0;
}

audio:not([controls]) {
    display: none;
}

[hidden] {
    display: none;
}

::-moz-selection {
    background: #4a89dc;
    color: #fff;
    text-shadow: none;
}

*.in-text-selection,
::selection {
    background: #4a89dc;
    color: #fff;
    text-shadow: none;
}

ul,
ol {
    padding: 0 0 0 1.875rem;
    /* 30px */
}

ul {
    list-style: square;
}

ol {
    list-style: decimal;
}

ul ul,
ol ol,
ul ol,
ol ul {
    margin: 0;
}

b,
th,
dt,
strong {
    font-weight: bold;
}

i,
em,
dfn,
cite {
    font-style: italic;
}

blockquote {
    padding-left: 1.875rem;
    margin: 0 0 1.875rem 1.875rem;
    border-left: solid 2px #474d54;
    padding-left: 30px;
    margin-top: 35px;
}

pre,
code,
kbd,
tt,
var {
    font-size: 0.875rem;
    font-family: Monaco, Consolas, "Andale Mono", "DejaVu Sans Mono", monospace;
}

code,
tt,
var {
    background: rgba(0, 0, 0, 0.05);
}

kbd {
    padding: 2px 4px;
    font-size: 90%;
    color: #fff;
    background-color: #333;
    border-radius: 3px;
    box-shadow: inset 0 -1px 0 rgba(0,0,0,.25);
}

pre.md-fences {
    padding: 10px 10px 10px 30px;
    margin-bottom: 20px;
    background: #333;
}

.CodeMirror-gutters {
    background: #333;
    border-right: 1px solid transparent;
}

.enable-diagrams pre.md-fences[lang="sequence"] .code-tooltip,
.enable-diagrams pre.md-fences[lang="flow"] .code-tooltip,
.enable-diagrams pre.md-fences[lang="mermaid"] .code-tooltip {
    bottom: -2.2em;
    right: 4px;
}

code,
kbd,
tt,
var {
    padding: 2px 5px;
}

table {
    max-width: 100%;
    width: 100%;
    border-collapse: collapse;
    border-spacing: 0;
}

th,
td {
    padding: 5px 10px;
    vertical-align: top;
}

a {
    -webkit-transition: all .2s ease-in-out;
    transition: all .2s ease-in-out;
}

hr {
    background: #474d54;
    /* variable */
}

h1 {
    margin-top: 2em;
}

a {
    color: #e0e0e0;
    text-decoration: underline;
}

a:hover {
    color: #fff;
}

.md-inline-math script {
    color: #81b1db;
}

b,
th,
dt,
strong {
    color: #DEDEDE;
    /* variable */
}

mark {
    background: #D3D40E;
}

blockquote {
    color: #9DA2A6;
}

table a {
    color: #DEDEDE;
    /* variable */
}

th,
td {
    border: solid 1px #474d54;
    /* variable */
}

.task-list {
    padding-left: 0;
}

.md-task-list-item {
    padding-left: 1.25rem;
}

.md-task-list-item > input {
    top: auto;
}

.md-task-list-item > input:before {
    content: "";
    display: inline-block;
    width: 0.875rem;
    height: 0.875rem;
    vertical-align: middle;
    text-align: center;
    border: 1px solid #b8bfc6;
    background-color: #363B40;
    margin-top: -0.4rem;
}

.md-task-list-item > input:checked:before,
.md-task-list-item > input[checked]:before {
    content: '\221A';
    /*◘*/
    font-size: 0.625rem;
    line-height: 0.625rem;
    color: #DEDEDE;
}

/** quick open **/
.auto-suggest-container {
    border: 0px;
    background-color: #525C65;
}

#typora-quick-open {
    background-color: #525C65;
}

#typora-quick-open input{
    background-color: #525C65;
    border: 0;
    border-bottom: 1px solid grey;
}

.typora-quick-open-item {
    background-color: inherit;
    color: inherit;
}

.typora-quick-open-item.active,
.typora-quick-open-item:hover {
    background-color: #4D8BDB;
    color: white;
}

.typora-quick-open-item:hover {
    background-color: rgba(77, 139, 219, 0.8);
}

.typora-search-spinner > div {
  background-color: #fff;
}

#write pre.md-meta-block {
    border-bottom: 1px dashed #ccc;
    background: transparent;
    padding-bottom: 0.6em;
    line-height: 1.6em;
}

.btn,
.btn .btn-default {
    background: transparent;
    color: #b8bfc6;
}

.ty-table-edit {
    border-top: 1px solid gray;
    background-color: #363B40;
}

.popover-title {
    background: transparent;
}

.md-image>.md-meta {
    color: #BBBBBB;
    background: transparent;
}

.md-expand.md-image>.md-meta {
    color: #DDD;
}

#write>h3:before,
#write>h4:before,
#write>h5:before,
#write>h6:before {
    border: none;
    border-radius: 0px;
    color: #888;
    text-decoration: underline;
    left: -1.4rem;
    top: 0.2rem;
}

#write>h3.md-focus:before {
    top: 2px;
}

#write>h4.md-focus:before {
    top: 2px;
}

.md-toc-item {
    color: #A8C2DC;
}

#write div.md-toc-tooltip {
    background-color: #363B40;
}

.dropdown-menu .btn:hover,
.dropdown-menu .btn:focus,
.md-toc .btn:hover,
.md-toc .btn:focus {
    color: white;
    background: black;
}

#toc-dropmenu {
    background: rgba(50, 54, 59, 0.93);
    border: 1px solid rgba(253, 253, 253, 0.15);
}

#toc-dropmenu .divider {
    background-color: #9b9b9b;
}

.outline-expander:before {
    top: 2px;
}

#typora-sidebar {
    box-shadow: none;
    border-right: 1px dashed;
    border-right: none;
}

.sidebar-tabs {
    border-bottom:0;
}

#typora-sidebar:hover .outline-title-wrapper {
    border-left: 1px dashed;
}

.outline-title-wrapper .btn {
    color: inherit;
}

.outline-item:hover {
    border-color: #363B40;
    background-color: #363B40;
    color: white;
}

h1.md-focus .md-attr,
h2.md-focus .md-attr,
h3.md-focus .md-attr,
h4.md-focus .md-attr,
h5.md-focus .md-attr,
h6.md-focus .md-attr,
.md-header-span .md-attr {
    color: #8C8E92;
    display: inline;
}

.md-comment {
    color: #5a95e3;
    opacity: 1;
}

.md-inline-math svg {
    color: #b8bfc6;
}

#math-inline-preview .md-arrow:after {
    background: black;
}

.modal-content {
    background: var(--bg-color);
    border: 0;
}

.modal-title {
    font-size: 1.5em;
}

.modal-content input {
    background-color: rgba(26, 21, 21, 0.51);
    color: white;
}

.modal-content .input-group-addon {
    color: white;
}

.modal-backdrop {
    background-color: rgba(174, 174, 174, 0.7);
}

.modal-content .btn-primary {
    border-color: var(--primary-color);
}

.md-table-resize-popover {
    background-color: #333;
}

.form-inline .input-group .input-group-addon {
    color: white;
}

#md-searchpanel {
    border-bottom: 1px dashed grey;
}

/** UI for electron */

.context-menu,
#spell-check-panel,
#footer-word-count-info {
    background-color: #42464A;
}

.context-menu.dropdown-menu .divider,
.dropdown-menu .divider {
    background-color: #777777;
}

footer {
    color: inherit;
}

@media (max-width: 1000px) {
    footer {
        border-top: none;
    }
    footer:hover {
        color: inherit;
    }
}

#file-info-file-path .file-info-field-value:hover {
    background-color: #555;
    color: #dedede;
}

.megamenu-content,
.megamenu-opened header {
    background: var(--bg-color);
}

.megamenu-menu-panel h2,
.megamenu-menu-panel h1,
.long-btn {
    color: inherit;
}

.megamenu-menu-panel input[type='text'] {
    background: inherit;
    border: 0;
    border-bottom: 1px solid;
}

#recent-file-panel-action-btn {
    background: inherit;
    border: 1px grey solid;
}

.megamenu-menu-panel .dropdown-menu > li > a {
    color: inherit;
    background-color: #2F353A;
    text-decoration: none;
}

.megamenu-menu-panel table td:nth-child(1) {
    color: inherit;
    font-weight: bold;
}

.megamenu-menu-panel tbody tr:hover td:nth-child(1) {
    color: white;
}

.modal-footer .btn-default, 
.modal-footer .btn-primary,
.modal-footer .btn-default:not(:hover) {
    border: 1px solid;
    border-color: transparent;
}

.btn-default:hover, .btn-default:focus, .btn-default.focus, .btn-default:active, .btn-default.active, .open > .dropdown-toggle.btn-default {
    color: white;
    border: 1px solid #ddd;
    background-color: inherit;
}

.modal-header {
    border-bottom: 0;
}

.modal-footer {
    border-top: 0;
}

#recent-file-panel tbody tr:nth-child(2n-1) {
    background-color: transparent !important;
}

.megamenu-menu-panel tbody tr:hover td:nth-child(2) {
    color: inherit;
}

.megamenu-menu-panel .btn {
    border: 1px solid #eee;
    background: transparent;
}

.mouse-hover .toolbar-icon.btn:hover,
#w-full.mouse-hover,
#w-pin.mouse-hover {
    background-color: inherit;
}

.typora-node::-webkit-scrollbar {
    width: 5px;
}

.typora-node::-webkit-scrollbar-thumb:vertical {
    background: rgba(250, 250, 250, 0.3);
}

.typora-node::-webkit-scrollbar-thumb:vertical:active {
    background: rgba(250, 250, 250, 0.5);
}

#w-unpin {
    background-color: #4182c4;
}

#top-titlebar, #top-titlebar * {
    color: var(--item-hover-text-color);
}

.typora-sourceview-on #toggle-sourceview-btn,
#footer-word-count:hover,
.ty-show-word-count #footer-word-count {
    background: #333333;
}

#toggle-sourceview-btn:hover {
    color: #eee;
    background: #333333;
}

/** focus mode */
.on-focus-mode .md-end-block:not(.md-focus):not(.md-focus-container) * {
    color: #686868 !important;
}

.on-focus-mode .md-end-block:not(.md-focus) img,
.on-focus-mode .md-task-list-item:not(.md-focus-container)>input {
    opacity: #686868 !important;
}

.on-focus-mode li[cid]:not(.md-focus-container){
    color: #686868;
}

.on-focus-mode .md-fences.md-focus .CodeMirror-code>*:not(.CodeMirror-activeline) *,
.on-focus-mode .CodeMirror.cm-s-inner:not(.CodeMirror-focused) * {
    color: #686868 !important;
}

.on-focus-mode .md-focus,
.on-focus-mode .md-focus-container {
    color: #fff;
}

.on-focus-mode #typora-source .CodeMirror-code>*:not(.CodeMirror-activeline) * {
    color: #686868 !important;
}


/*diagrams*/
#write .md-focus .md-diagram-panel {
    border: 1px solid #ddd;
    margin-left: -1px;
    width: calc(100% + 2px);
}

/*diagrams*/
#write .md-focus.md-fences-with-lineno .md-diagram-panel {
    margin-left: auto;
}

.md-diagram-panel-error {
    color: #f1908e;
}

.active-tab-files #info-panel-tab-file,
.active-tab-files #info-panel-tab-file:hover,
.active-tab-outline #info-panel-tab-outline,
.active-tab-outline #info-panel-tab-outline:hover {
    color: #eee;
}

.sidebar-footer-item:hover,
.footer-item:hover {
    background: inherit;
    color: white;
}

.ty-side-sort-btn.active,
.ty-side-sort-btn:hover,
.selected-folder-menu-item a:after {
    color: white;
}

#sidebar-files-menu {
    border:solid 1px;
    box-shadow: 4px 4px 20px rgba(0, 0, 0, 0.79);
    background-color: var(--bg-color);
}

.file-list-item {
    border-bottom:none;
}

.file-list-item-summary {
    opacity: 1;
}

.file-list-item.active:first-child {
    border-top: none;
}

.file-node-background {
    height: 32px;
}

.file-library-node.active>.file-node-content,
.file-list-item.active {
    color: white;
    color: var(--active-file-text-color);
}

.file-library-node.active>.file-node-background{
    background-color: rgb(34, 34, 34);
    background-color: var(--active-file-bg-color);
}
.file-list-item.active {
    background-color: rgb(34, 34, 34);
    background-color: var(--active-file-bg-color);
}

#ty-tooltip {
    background-color: black;
    color: #eee;
}

.md-task-list-item>input {
    margin-left: -1.3em;
    margin-top: 0.3rem;
    -webkit-appearance: none;
}

.md-mathjax-midline {
    background-color: #57616b;
    border-bottom: none;
}

footer.ty-footer {
    border-color: #656565;
}

.ty-preferences .btn-default {
    background: transparent;
}
.ty-preferences .btn-default:hover {
    background: #57616b;
}

.ty-preferences select {
    border: 1px solid #989698;
    height: 21px;
}

.ty-preferences .nav-group-item.active {
    background: var(--item-hover-bg-color);
}

.ty-preferences input[type="search"] {
    border-color: #333;
    background: #333;
    line-height: 22px;
    border-radius: 6px;
    color: white;
}

.ty-preferences input[type="search"]:focus {
    box-shadow: none;
}

[data-is-directory="true"] .file-node-content {
    margin-bottom: 0;
}

.file-node-title {
    line-height: 22px;
}

.html-for-mac .file-node-open-state, .html-for-mac .file-node-icon {
    line-height: 26px;
}

::-webkit-scrollbar-thumb {
    background: rgba(230, 230, 230, 0.30);
}

::-webkit-scrollbar-thumb:active {
    background: rgba(230, 230, 230, 0.50);
}

#typora-sidebar:hover div.sidebar-content-content::-webkit-scrollbar-thumb:horizontal {
    background: rgba(230, 230, 230, 0.30);
}

.nav-group-item:active {
    background-color: #474d54;
}

.md-search-hit {
    background: rgba(199, 140, 60, 0.81);
    color: #eee;
}

.md-search-hit * {
    color: #eee;
}

#md-searchpanel input {
    color: white;
}

.export-detail,
.export-item.active,
.export-items-list-control {
    background: #d6d6d4
}


</style>
</head>
<body class='typora-export os-windows'>
<div id='write'  class=''><h2><a name="第六章函数近似function-approximation）方法" class="md-header-anchor"></a><span>第六章：函数近似（function approximation）方法</span></h2><p><span>在有些任务中，状态和动作对的数目非常大，甚至可能是无穷大，这时不可能对所有状态（或状态动作对）逐一进行更新。函数近似方法用参数化的模型来近似整个状态价值函数（或动作价值函数），并在每次学习时更新整个函数。</span></p><h3><a name="一函数近似原理" class="md-header-anchor"></a><span>一、函数近似原理</span></h3><p><span>函数近似（function approximation）方法用带参数 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="1.93ex" height="1.36ex" viewBox="0 -500.4 831 585.5" role="img" focusable="false" style="vertical-align: -0.198ex;"><defs><path stroke-width="0" id="E337-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E337-MJMAINB-77" x="0" y="0"></use></g></svg></span><script type="math/tex">\bold w</script><span> 的函数来近似价值函数，如用 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="14.083ex" height="2.71ex" viewBox="0 -832.7 6063.7 1166.9" role="img" focusable="false" style="vertical-align: -0.776ex;"><defs><path stroke-width="0" id="E291-MJMATHI-76" d="M173 380Q173 405 154 405Q130 405 104 376T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Q21 294 29 316T53 368T97 419T160 441Q202 441 225 417T249 361Q249 344 246 335Q246 329 231 291T200 202T182 113Q182 86 187 69Q200 26 250 26Q287 26 319 60T369 139T398 222T409 277Q409 300 401 317T383 343T365 361T357 383Q357 405 376 424T417 443Q436 443 451 425T467 367Q467 340 455 284T418 159T347 40T241 -11Q177 -11 139 22Q102 54 102 117Q102 148 110 181T151 298Q173 362 173 380Z"></path><path stroke-width="0" id="E291-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E291-MJMATHI-73" d="M131 289Q131 321 147 354T203 415T300 442Q362 442 390 415T419 355Q419 323 402 308T364 292Q351 292 340 300T328 326Q328 342 337 354T354 372T367 378Q368 378 368 379Q368 382 361 388T336 399T297 405Q249 405 227 379T204 326Q204 301 223 291T278 274T330 259Q396 230 396 163Q396 135 385 107T352 51T289 7T195 -10Q118 -10 86 19T53 87Q53 126 74 143T118 160Q133 160 146 151T160 120Q160 94 142 76T111 58Q109 57 108 57T107 55Q108 52 115 47T146 34T201 27Q237 27 263 38T301 66T318 97T323 122Q323 150 302 164T254 181T195 196T148 231Q131 256 131 289Z"></path><path stroke-width="0" id="E291-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E291-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E291-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E291-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E291-MJMAIN-2208" d="M84 250Q84 372 166 450T360 539Q361 539 377 539T419 540T469 540H568Q583 532 583 520Q583 511 570 501L466 500Q355 499 329 494Q280 482 242 458T183 409T147 354T129 306T124 272V270H568Q583 262 583 250T568 230H124V228Q124 207 134 177T167 112T231 48T328 7Q355 1 466 0H570Q583 -10 583 -20Q583 -32 568 -40H471Q464 -40 446 -40T417 -41Q262 -41 172 45Q84 127 84 250Z"></path><path stroke-width="0" id="E291-MJCAL-53" d="M554 512Q536 512 536 522Q536 525 539 539T542 564Q542 588 528 604Q515 616 482 625T410 635Q374 635 349 624T312 594T295 561T290 532Q290 505 303 482T342 442T378 419T409 404Q435 391 451 383T494 357T535 323T562 282T574 231Q574 133 464 56T220 -22Q138 -22 78 21T18 123Q18 184 61 227T156 274Q178 274 178 263Q178 260 177 258Q172 247 164 239T151 227T136 218L127 213L124 202Q118 186 118 163Q120 124 165 86T292 48Q374 48 423 86T473 186V193Q473 267 347 327Q268 364 239 389Q191 431 191 486Q191 547 242 600T356 679T470 705Q472 705 478 705T489 704Q551 704 596 682T642 610Q642 566 621 545Q592 516 554 512Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E291-MJMATHI-76" x="0" y="0"></use><use xlink:href="#E291-MJMAIN-28" x="485" y="0"></use><use xlink:href="#E291-MJMATHI-73" x="874" y="0"></use><use xlink:href="#E291-MJMAIN-3B" x="1343" y="0"></use><use xlink:href="#E291-MJMAINB-77" x="1787" y="0"></use><use xlink:href="#E291-MJMAIN-29" x="2618" y="0"></use><use xlink:href="#E291-MJMAIN-2C" x="3007" y="0"></use><use xlink:href="#E291-MJMATHI-73" x="3730" y="0"></use><use xlink:href="#E291-MJMAIN-2208" x="4476" y="0"></use><use xlink:href="#E291-MJCAL-53" x="5421" y="0"></use></g></svg></span><script type="math/tex">v(s;\bold w),\; s \in \mathcal S</script><span> 近似状态价值函数，用 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="23.29ex" height="2.71ex" viewBox="0 -832.7 10027.6 1166.9" role="img" focusable="false" style="vertical-align: -0.776ex;"><defs><path stroke-width="0" id="E292-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E292-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E292-MJMATHI-73" d="M131 289Q131 321 147 354T203 415T300 442Q362 442 390 415T419 355Q419 323 402 308T364 292Q351 292 340 300T328 326Q328 342 337 354T354 372T367 378Q368 378 368 379Q368 382 361 388T336 399T297 405Q249 405 227 379T204 326Q204 301 223 291T278 274T330 259Q396 230 396 163Q396 135 385 107T352 51T289 7T195 -10Q118 -10 86 19T53 87Q53 126 74 143T118 160Q133 160 146 151T160 120Q160 94 142 76T111 58Q109 57 108 57T107 55Q108 52 115 47T146 34T201 27Q237 27 263 38T301 66T318 97T323 122Q323 150 302 164T254 181T195 196T148 231Q131 256 131 289Z"></path><path stroke-width="0" id="E292-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E292-MJMATHI-61" d="M33 157Q33 258 109 349T280 441Q331 441 370 392Q386 422 416 422Q429 422 439 414T449 394Q449 381 412 234T374 68Q374 43 381 35T402 26Q411 27 422 35Q443 55 463 131Q469 151 473 152Q475 153 483 153H487Q506 153 506 144Q506 138 501 117T481 63T449 13Q436 0 417 -8Q409 -10 393 -10Q359 -10 336 5T306 36L300 51Q299 52 296 50Q294 48 292 46Q233 -10 172 -10Q117 -10 75 30T33 157ZM351 328Q351 334 346 350T323 385T277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q217 26 254 59T298 110Q300 114 325 217T351 328Z"></path><path stroke-width="0" id="E292-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E292-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E292-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E292-MJMAIN-2208" d="M84 250Q84 372 166 450T360 539Q361 539 377 539T419 540T469 540H568Q583 532 583 520Q583 511 570 501L466 500Q355 499 329 494Q280 482 242 458T183 409T147 354T129 306T124 272V270H568Q583 262 583 250T568 230H124V228Q124 207 134 177T167 112T231 48T328 7Q355 1 466 0H570Q583 -10 583 -20Q583 -32 568 -40H471Q464 -40 446 -40T417 -41Q262 -41 172 45Q84 127 84 250Z"></path><path stroke-width="0" id="E292-MJCAL-53" d="M554 512Q536 512 536 522Q536 525 539 539T542 564Q542 588 528 604Q515 616 482 625T410 635Q374 635 349 624T312 594T295 561T290 532Q290 505 303 482T342 442T378 419T409 404Q435 391 451 383T494 357T535 323T562 282T574 231Q574 133 464 56T220 -22Q138 -22 78 21T18 123Q18 184 61 227T156 274Q178 274 178 263Q178 260 177 258Q172 247 164 239T151 227T136 218L127 213L124 202Q118 186 118 163Q120 124 165 86T292 48Q374 48 423 86T473 186V193Q473 267 347 327Q268 364 239 389Q191 431 191 486Q191 547 242 600T356 679T470 705Q472 705 478 705T489 704Q551 704 596 682T642 610Q642 566 621 545Q592 516 554 512Z"></path><path stroke-width="0" id="E292-MJCAL-41" d="M576 668Q576 688 606 708T660 728Q676 728 675 712V571Q675 409 688 252Q696 122 720 57Q722 53 723 50T728 46T732 43T737 41T743 39L754 45Q788 61 803 61Q819 61 819 47Q818 43 814 35Q799 15 755 -7T675 -30Q659 -30 648 -25T630 -8T621 11T614 34Q603 77 599 106T594 146T591 160V163H460L329 164L316 145Q241 35 196 -7T119 -50T59 -24T30 43Q30 75 46 100T74 125Q81 125 83 120T88 104T96 84Q118 57 151 57Q189 57 277 182Q432 400 542 625L559 659H567Q574 659 575 660T576 668ZM584 249Q579 333 577 386T575 473T574 520V581L563 560Q497 426 412 290L372 228L370 224H371L383 228L393 232H586L584 249Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E292-MJMATHI-71" x="0" y="0"></use><use xlink:href="#E292-MJMAIN-28" x="460" y="0"></use><use xlink:href="#E292-MJMATHI-73" x="849" y="0"></use><use xlink:href="#E292-MJMAIN-2C" x="1318" y="0"></use><use xlink:href="#E292-MJMATHI-61" x="1762" y="0"></use><use xlink:href="#E292-MJMAIN-3B" x="2291" y="0"></use><use xlink:href="#E292-MJMAINB-77" x="2736" y="0"></use><use xlink:href="#E292-MJMAIN-29" x="3567" y="0"></use><use xlink:href="#E292-MJMAIN-2C" x="3956" y="0"></use><use xlink:href="#E292-MJMATHI-73" x="4678" y="0"></use><use xlink:href="#E292-MJMAIN-2208" x="5425" y="0"></use><use xlink:href="#E292-MJCAL-53" x="6370" y="0"></use><use xlink:href="#E292-MJMAIN-2C" x="7012" y="0"></use><use xlink:href="#E292-MJMATHI-61" x="7457" y="0"></use><use xlink:href="#E292-MJMAIN-2208" x="8263" y="0"></use><use xlink:href="#E292-MJCAL-41" x="9208" y="0"></use></g></svg></span><script type="math/tex">q(s,a;\bold w),\; s \in \mathcal S,a \in \mathcal A</script><span> 近似动作价值函数。当动作集有限时，还能用矢量函数 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="36.025ex" height="2.71ex" viewBox="0 -832.7 15510.7 1166.9" role="img" focusable="false" style="vertical-align: -0.776ex;"><defs><path stroke-width="0" id="E293-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E293-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E293-MJMATHI-73" d="M131 289Q131 321 147 354T203 415T300 442Q362 442 390 415T419 355Q419 323 402 308T364 292Q351 292 340 300T328 326Q328 342 337 354T354 372T367 378Q368 378 368 379Q368 382 361 388T336 399T297 405Q249 405 227 379T204 326Q204 301 223 291T278 274T330 259Q396 230 396 163Q396 135 385 107T352 51T289 7T195 -10Q118 -10 86 19T53 87Q53 126 74 143T118 160Q133 160 146 151T160 120Q160 94 142 76T111 58Q109 57 108 57T107 55Q108 52 115 47T146 34T201 27Q237 27 263 38T301 66T318 97T323 122Q323 150 302 164T254 181T195 196T148 231Q131 256 131 289Z"></path><path stroke-width="0" id="E293-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E293-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E293-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E293-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E293-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E293-MJMATHI-61" d="M33 157Q33 258 109 349T280 441Q331 441 370 392Q386 422 416 422Q429 422 439 414T449 394Q449 381 412 234T374 68Q374 43 381 35T402 26Q411 27 422 35Q443 55 463 131Q469 151 473 152Q475 153 483 153H487Q506 153 506 144Q506 138 501 117T481 63T449 13Q436 0 417 -8Q409 -10 393 -10Q359 -10 336 5T306 36L300 51Q299 52 296 50Q294 48 292 46Q233 -10 172 -10Q117 -10 75 30T33 157ZM351 328Q351 334 346 350T323 385T277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q217 26 254 59T298 110Q300 114 325 217T351 328Z"></path><path stroke-width="0" id="E293-MJMAIN-3A" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 84 95 102T138 120Q162 120 180 104T199 61Q199 36 182 18T139 0T96 17T78 60Z"></path><path stroke-width="0" id="E293-MJMAIN-2208" d="M84 250Q84 372 166 450T360 539Q361 539 377 539T419 540T469 540H568Q583 532 583 520Q583 511 570 501L466 500Q355 499 329 494Q280 482 242 458T183 409T147 354T129 306T124 272V270H568Q583 262 583 250T568 230H124V228Q124 207 134 177T167 112T231 48T328 7Q355 1 466 0H570Q583 -10 583 -20Q583 -32 568 -40H471Q464 -40 446 -40T417 -41Q262 -41 172 45Q84 127 84 250Z"></path><path stroke-width="0" id="E293-MJCAL-41" d="M576 668Q576 688 606 708T660 728Q676 728 675 712V571Q675 409 688 252Q696 122 720 57Q722 53 723 50T728 46T732 43T737 41T743 39L754 45Q788 61 803 61Q819 61 819 47Q818 43 814 35Q799 15 755 -7T675 -30Q659 -30 648 -25T630 -8T621 11T614 34Q603 77 599 106T594 146T591 160V163H460L329 164L316 145Q241 35 196 -7T119 -50T59 -24T30 43Q30 75 46 100T74 125Q81 125 83 120T88 104T96 84Q118 57 151 57Q189 57 277 182Q432 400 542 625L559 659H567Q574 659 575 660T576 668ZM584 249Q579 333 577 386T575 473T574 520V581L563 560Q497 426 412 290L372 228L370 224H371L383 228L393 232H586L584 249Z"></path><path stroke-width="0" id="E293-MJCAL-53" d="M554 512Q536 512 536 522Q536 525 539 539T542 564Q542 588 528 604Q515 616 482 625T410 635Q374 635 349 624T312 594T295 561T290 532Q290 505 303 482T342 442T378 419T409 404Q435 391 451 383T494 357T535 323T562 282T574 231Q574 133 464 56T220 -22Q138 -22 78 21T18 123Q18 184 61 227T156 274Q178 274 178 263Q178 260 177 258Q172 247 164 239T151 227T136 218L127 213L124 202Q118 186 118 163Q120 124 165 86T292 48Q374 48 423 86T473 186V193Q473 267 347 327Q268 364 239 389Q191 431 191 486Q191 547 242 600T356 679T470 705Q472 705 478 705T489 704Q551 704 596 682T642 610Q642 566 621 545Q592 516 554 512Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E293-MJMATHI-71" x="0" y="0"></use><use xlink:href="#E293-MJMAIN-28" x="460" y="0"></use><use xlink:href="#E293-MJMATHI-73" x="849" y="0"></use><use xlink:href="#E293-MJMAIN-3B" x="1318" y="0"></use><use xlink:href="#E293-MJMAINB-77" x="1762" y="0"></use><use xlink:href="#E293-MJMAIN-29" x="2593" y="0"></use><use xlink:href="#E293-MJMAIN-3D" x="3260" y="0"></use><use xlink:href="#E293-MJMAIN-28" x="4316" y="0"></use><use xlink:href="#E293-MJMATHI-71" x="4705" y="0"></use><use xlink:href="#E293-MJMAIN-28" x="5165" y="0"></use><use xlink:href="#E293-MJMATHI-73" x="5554" y="0"></use><use xlink:href="#E293-MJMAIN-2C" x="6023" y="0"></use><use xlink:href="#E293-MJMATHI-61" x="6467" y="0"></use><use xlink:href="#E293-MJMAIN-3B" x="6996" y="0"></use><use xlink:href="#E293-MJMAINB-77" x="7441" y="0"></use><use xlink:href="#E293-MJMAIN-29" x="8272" y="0"></use><use xlink:href="#E293-MJMAIN-3A" x="8939" y="0"></use><use xlink:href="#E293-MJMATHI-61" x="9495" y="0"></use><use xlink:href="#E293-MJMAIN-2208" x="10301" y="0"></use><use xlink:href="#E293-MJCAL-41" x="11246" y="0"></use><use xlink:href="#E293-MJMAIN-29" x="12065" y="0"></use><use xlink:href="#E293-MJMAIN-2C" x="12454" y="0"></use><use xlink:href="#E293-MJMATHI-73" x="13177" y="0"></use><use xlink:href="#E293-MJMAIN-2208" x="13923" y="0"></use><use xlink:href="#E293-MJCAL-53" x="14868" y="0"></use></g></svg></span><script type="math/tex">q(s;\bold w)=(q(s,a;\bold w):a \in \mathcal A),\; s \in \mathcal S</script><span> 来近似动作价值，矢量函数 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="6.928ex" height="2.71ex" viewBox="0 -832.7 2982.7 1166.9" role="img" focusable="false" style="vertical-align: -0.776ex;"><defs><path stroke-width="0" id="E294-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E294-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E294-MJMATHI-73" d="M131 289Q131 321 147 354T203 415T300 442Q362 442 390 415T419 355Q419 323 402 308T364 292Q351 292 340 300T328 326Q328 342 337 354T354 372T367 378Q368 378 368 379Q368 382 361 388T336 399T297 405Q249 405 227 379T204 326Q204 301 223 291T278 274T330 259Q396 230 396 163Q396 135 385 107T352 51T289 7T195 -10Q118 -10 86 19T53 87Q53 126 74 143T118 160Q133 160 146 151T160 120Q160 94 142 76T111 58Q109 57 108 57T107 55Q108 52 115 47T146 34T201 27Q237 27 263 38T301 66T318 97T323 122Q323 150 302 164T254 181T195 196T148 231Q131 256 131 289Z"></path><path stroke-width="0" id="E294-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E294-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E294-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E294-MJMATHI-71" x="0" y="0"></use><use xlink:href="#E294-MJMAIN-28" x="460" y="0"></use><use xlink:href="#E294-MJMATHI-73" x="849" y="0"></use><use xlink:href="#E294-MJMAIN-3B" x="1318" y="0"></use><use xlink:href="#E294-MJMAINB-77" x="1762" y="0"></use><use xlink:href="#E294-MJMAIN-29" x="2593" y="0"></use></g></svg></span><script type="math/tex">q(s;\bold w)</script><span> 的每一个元素对应着一个动作，而整个矢量函数除参数外只用状态作为输入。</span></p><p><span>函数近似方法可以使用随机梯度下降算法或者半梯度下降算法对价值函数进行更新。以动作价值更新为例，</span><strong><span>随机梯度下降</span></strong><span>（stochastic gradient-descent, SGD）算法就是在试图减小每一步的回报估计 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="2.651ex" height="2.228ex" viewBox="0 -749.6 1141.3 959.2" role="img" focusable="false" style="vertical-align: -0.487ex;"><defs><path stroke-width="0" id="E295-MJMATHI-47" d="M50 252Q50 367 117 473T286 641T490 704Q580 704 633 653Q642 643 648 636T656 626L657 623Q660 623 684 649Q691 655 699 663T715 679T725 690L740 705H746Q760 705 760 698Q760 694 728 561Q692 422 692 421Q690 416 687 415T669 413H653Q647 419 647 422Q647 423 648 429T650 449T651 481Q651 552 619 605T510 659Q492 659 471 656T418 643T357 615T294 567T236 496T189 394T158 260Q156 242 156 221Q156 173 170 136T206 79T256 45T308 28T353 24Q407 24 452 47T514 106Q517 114 529 161T541 214Q541 222 528 224T468 227H431Q425 233 425 235T427 254Q431 267 437 273H454Q494 271 594 271Q634 271 659 271T695 272T707 272Q721 272 721 263Q721 261 719 249Q714 230 709 228Q706 227 694 227Q674 227 653 224Q646 221 643 215T629 164Q620 131 614 108Q589 6 586 3Q584 1 581 1Q571 1 553 21T530 52Q530 53 528 52T522 47Q448 -22 322 -22Q201 -22 126 55T50 252Z"></path><path stroke-width="0" id="E295-MJMATHI-74" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E295-MJMATHI-47" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E295-MJMATHI-74" x="1111" y="-213"></use></g></svg></span><script type="math/tex">G_t</script><span> 和动作价值 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="11.687ex" height="2.71ex" viewBox="0 -832.7 5031.9 1166.9" role="img" focusable="false" style="vertical-align: -0.776ex;"><defs><path stroke-width="0" id="E296-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E296-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E296-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E296-MJMATHI-74" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path><path stroke-width="0" id="E296-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E296-MJMATHI-41" d="M208 74Q208 50 254 46Q272 46 272 35Q272 34 270 22Q267 8 264 4T251 0Q249 0 239 0T205 1T141 2Q70 2 50 0H42Q35 7 35 11Q37 38 48 46H62Q132 49 164 96Q170 102 345 401T523 704Q530 716 547 716H555H572Q578 707 578 706L606 383Q634 60 636 57Q641 46 701 46Q726 46 726 36Q726 34 723 22Q720 7 718 4T704 0Q701 0 690 0T651 1T578 2Q484 2 455 0H443Q437 6 437 9T439 27Q443 40 445 43L449 46H469Q523 49 533 63L521 213H283L249 155Q208 86 208 74ZM516 260Q516 271 504 416T490 562L463 519Q447 492 400 412L310 260L413 259Q516 259 516 260Z"></path><path stroke-width="0" id="E296-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E296-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E296-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E296-MJMATHI-71" x="0" y="0"></use><use xlink:href="#E296-MJMAIN-28" x="460" y="0"></use><g transform="translate(849,0)"><use xlink:href="#E296-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E296-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E296-MJMAIN-2C" x="1817" y="0"></use><g transform="translate(2261,0)"><use xlink:href="#E296-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E296-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E296-MJMAIN-3B" x="3367" y="0"></use><use xlink:href="#E296-MJMAINB-77" x="3811" y="0"></use><use xlink:href="#E296-MJMAIN-29" x="4642" y="0"></use></g></svg></span><script type="math/tex">q(S_t,A_t;\bold w)</script><span> 的差别时，定义每一步损失为 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="19.522ex" height="2.903ex" viewBox="0 -915.7 8405.1 1250" role="img" focusable="false" style="vertical-align: -0.776ex;"><defs><path stroke-width="0" id="E297-MJMAIN-5B" d="M118 -250V750H255V710H158V-210H255V-250H118Z"></path><path stroke-width="0" id="E297-MJMATHI-47" d="M50 252Q50 367 117 473T286 641T490 704Q580 704 633 653Q642 643 648 636T656 626L657 623Q660 623 684 649Q691 655 699 663T715 679T725 690L740 705H746Q760 705 760 698Q760 694 728 561Q692 422 692 421Q690 416 687 415T669 413H653Q647 419 647 422Q647 423 648 429T650 449T651 481Q651 552 619 605T510 659Q492 659 471 656T418 643T357 615T294 567T236 496T189 394T158 260Q156 242 156 221Q156 173 170 136T206 79T256 45T308 28T353 24Q407 24 452 47T514 106Q517 114 529 161T541 214Q541 222 528 224T468 227H431Q425 233 425 235T427 254Q431 267 437 273H454Q494 271 594 271Q634 271 659 271T695 272T707 272Q721 272 721 263Q721 261 719 249Q714 230 709 228Q706 227 694 227Q674 227 653 224Q646 221 643 215T629 164Q620 131 614 108Q589 6 586 3Q584 1 581 1Q571 1 553 21T530 52Q530 53 528 52T522 47Q448 -22 322 -22Q201 -22 126 55T50 252Z"></path><path stroke-width="0" id="E297-MJMATHI-74" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path><path stroke-width="0" id="E297-MJMAIN-2212" d="M84 237T84 250T98 270H679Q694 262 694 250T679 230H98Q84 237 84 250Z"></path><path stroke-width="0" id="E297-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E297-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E297-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E297-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E297-MJMATHI-41" d="M208 74Q208 50 254 46Q272 46 272 35Q272 34 270 22Q267 8 264 4T251 0Q249 0 239 0T205 1T141 2Q70 2 50 0H42Q35 7 35 11Q37 38 48 46H62Q132 49 164 96Q170 102 345 401T523 704Q530 716 547 716H555H572Q578 707 578 706L606 383Q634 60 636 57Q641 46 701 46Q726 46 726 36Q726 34 723 22Q720 7 718 4T704 0Q701 0 690 0T651 1T578 2Q484 2 455 0H443Q437 6 437 9T439 27Q443 40 445 43L449 46H469Q523 49 533 63L521 213H283L249 155Q208 86 208 74ZM516 260Q516 271 504 416T490 562L463 519Q447 492 400 412L310 260L413 259Q516 259 516 260Z"></path><path stroke-width="0" id="E297-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E297-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E297-MJMAIN-5D" d="M22 710V750H159V-250H22V-210H119V710H22Z"></path><path stroke-width="0" id="E297-MJMAIN-32" d="M109 429Q82 429 66 447T50 491Q50 562 103 614T235 666Q326 666 387 610T449 465Q449 422 429 383T381 315T301 241Q265 210 201 149L142 93L218 92Q375 92 385 97Q392 99 409 186V189H449V186Q448 183 436 95T421 3V0H50V19V31Q50 38 56 46T86 81Q115 113 136 137Q145 147 170 174T204 211T233 244T261 278T284 308T305 340T320 369T333 401T340 431T343 464Q343 527 309 573T212 619Q179 619 154 602T119 569T109 550Q109 549 114 549Q132 549 151 535T170 489Q170 464 154 447T109 429Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E297-MJMAIN-5B" x="0" y="0"></use><g transform="translate(278,0)"><use xlink:href="#E297-MJMATHI-47" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E297-MJMATHI-74" x="1111" y="-213"></use></g><use xlink:href="#E297-MJMAIN-2212" x="1641" y="0"></use><use xlink:href="#E297-MJMATHI-71" x="2641" y="0"></use><use xlink:href="#E297-MJMAIN-28" x="3101" y="0"></use><g transform="translate(3490,0)"><use xlink:href="#E297-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E297-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E297-MJMAIN-2C" x="4458" y="0"></use><g transform="translate(4903,0)"><use xlink:href="#E297-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E297-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E297-MJMAIN-2C" x="6008" y="0"></use><use xlink:href="#E297-MJMAINB-77" x="6453" y="0"></use><use xlink:href="#E297-MJMAIN-29" x="7284" y="0"></use><g transform="translate(7673,0)"><use xlink:href="#E297-MJMAIN-5D" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E297-MJMAIN-32" x="393" y="513"></use></g></g></svg></span><script type="math/tex">[G_t-q(S_t,A_t,\bold w)]^2</script><span> ，那么对整个回合的损失函数为 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="22.875ex" height="7.051ex" viewBox="0 -1787.9 9849.1 3035.9" role="img" focusable="false" style="vertical-align: -2.899ex;"><defs><path stroke-width="0" id="E298-MJSZ2-2211" d="M60 948Q63 950 665 950H1267L1325 815Q1384 677 1388 669H1348L1341 683Q1320 724 1285 761Q1235 809 1174 838T1033 881T882 898T699 902H574H543H251L259 891Q722 258 724 252Q725 250 724 246Q721 243 460 -56L196 -356Q196 -357 407 -357Q459 -357 548 -357T676 -358Q812 -358 896 -353T1063 -332T1204 -283T1307 -196Q1328 -170 1348 -124H1388Q1388 -125 1381 -145T1356 -210T1325 -294L1267 -449L666 -450Q64 -450 61 -448Q55 -446 55 -439Q55 -437 57 -433L590 177Q590 178 557 222T452 366T322 544L56 909L55 924Q55 945 60 948Z"></path><path stroke-width="0" id="E298-MJMATHI-74" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path><path stroke-width="0" id="E298-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E298-MJMAIN-30" d="M96 585Q152 666 249 666Q297 666 345 640T423 548Q460 465 460 320Q460 165 417 83Q397 41 362 16T301 -15T250 -22Q224 -22 198 -16T137 16T82 83Q39 165 39 320Q39 494 96 585ZM321 597Q291 629 250 629Q208 629 178 597Q153 571 145 525T137 333Q137 175 145 125T181 46Q209 16 250 16Q290 16 318 46Q347 76 354 130T362 333Q362 478 354 524T321 597Z"></path><path stroke-width="0" id="E298-MJMATHI-54" d="M40 437Q21 437 21 445Q21 450 37 501T71 602L88 651Q93 669 101 677H569H659Q691 677 697 676T704 667Q704 661 687 553T668 444Q668 437 649 437Q640 437 637 437T631 442L629 445Q629 451 635 490T641 551Q641 586 628 604T573 629Q568 630 515 631Q469 631 457 630T439 622Q438 621 368 343T298 60Q298 48 386 46Q418 46 427 45T436 36Q436 31 433 22Q429 4 424 1L422 0Q419 0 415 0Q410 0 363 1T228 2Q99 2 64 0H49Q43 6 43 9T45 27Q49 40 55 46H83H94Q174 46 189 55Q190 56 191 56Q196 59 201 76T241 233Q258 301 269 344Q339 619 339 625Q339 630 310 630H279Q212 630 191 624Q146 614 121 583T67 467Q60 445 57 441T43 437H40Z"></path><path stroke-width="0" id="E298-MJMAIN-2212" d="M84 237T84 250T98 270H679Q694 262 694 250T679 230H98Q84 237 84 250Z"></path><path stroke-width="0" id="E298-MJMAIN-31" d="M213 578L200 573Q186 568 160 563T102 556H83V602H102Q149 604 189 617T245 641T273 663Q275 666 285 666Q294 666 302 660V361L303 61Q310 54 315 52T339 48T401 46H427V0H416Q395 3 257 3Q121 3 100 0H88V46H114Q136 46 152 46T177 47T193 50T201 52T207 57T213 61V578Z"></path><path stroke-width="0" id="E298-MJMAIN-5B" d="M118 -250V750H255V710H158V-210H255V-250H118Z"></path><path stroke-width="0" id="E298-MJMATHI-47" d="M50 252Q50 367 117 473T286 641T490 704Q580 704 633 653Q642 643 648 636T656 626L657 623Q660 623 684 649Q691 655 699 663T715 679T725 690L740 705H746Q760 705 760 698Q760 694 728 561Q692 422 692 421Q690 416 687 415T669 413H653Q647 419 647 422Q647 423 648 429T650 449T651 481Q651 552 619 605T510 659Q492 659 471 656T418 643T357 615T294 567T236 496T189 394T158 260Q156 242 156 221Q156 173 170 136T206 79T256 45T308 28T353 24Q407 24 452 47T514 106Q517 114 529 161T541 214Q541 222 528 224T468 227H431Q425 233 425 235T427 254Q431 267 437 273H454Q494 271 594 271Q634 271 659 271T695 272T707 272Q721 272 721 263Q721 261 719 249Q714 230 709 228Q706 227 694 227Q674 227 653 224Q646 221 643 215T629 164Q620 131 614 108Q589 6 586 3Q584 1 581 1Q571 1 553 21T530 52Q530 53 528 52T522 47Q448 -22 322 -22Q201 -22 126 55T50 252Z"></path><path stroke-width="0" id="E298-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E298-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E298-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E298-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E298-MJMATHI-41" d="M208 74Q208 50 254 46Q272 46 272 35Q272 34 270 22Q267 8 264 4T251 0Q249 0 239 0T205 1T141 2Q70 2 50 0H42Q35 7 35 11Q37 38 48 46H62Q132 49 164 96Q170 102 345 401T523 704Q530 716 547 716H555H572Q578 707 578 706L606 383Q634 60 636 57Q641 46 701 46Q726 46 726 36Q726 34 723 22Q720 7 718 4T704 0Q701 0 690 0T651 1T578 2Q484 2 455 0H443Q437 6 437 9T439 27Q443 40 445 43L449 46H469Q523 49 533 63L521 213H283L249 155Q208 86 208 74ZM516 260Q516 271 504 416T490 562L463 519Q447 492 400 412L310 260L413 259Q516 259 516 260Z"></path><path stroke-width="0" id="E298-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E298-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E298-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E298-MJMAIN-5D" d="M22 710V750H159V-250H22V-210H119V710H22Z"></path><path stroke-width="0" id="E298-MJMAIN-32" d="M109 429Q82 429 66 447T50 491Q50 562 103 614T235 666Q326 666 387 610T449 465Q449 422 429 383T381 315T301 241Q265 210 201 149L142 93L218 92Q375 92 385 97Q392 99 409 186V189H449V186Q448 183 436 95T421 3V0H50V19V31Q50 38 56 46T86 81Q115 113 136 137Q145 147 170 174T204 211T233 244T261 278T284 308T305 340T320 369T333 401T340 431T343 464Q343 527 309 573T212 619Q179 619 154 602T119 569T109 550Q109 549 114 549Q132 549 151 535T170 489Q170 464 154 447T109 429Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E298-MJSZ2-2211" x="0" y="0"></use><g transform="translate(142,-1088)"><use transform="scale(0.707)" xlink:href="#E298-MJMATHI-74" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E298-MJMAIN-3D" x="361" y="0"></use><use transform="scale(0.707)" xlink:href="#E298-MJMAIN-30" x="1139" y="0"></use></g><g transform="translate(21,1150)"><use transform="scale(0.707)" xlink:href="#E298-MJMATHI-54" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E298-MJMAIN-2212" x="704" y="0"></use><use transform="scale(0.707)" xlink:href="#E298-MJMAIN-31" x="1482" y="0"></use></g><use xlink:href="#E298-MJMAIN-5B" x="1444" y="0"></use><g transform="translate(1722,0)"><use xlink:href="#E298-MJMATHI-47" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E298-MJMATHI-74" x="1111" y="-213"></use></g><use xlink:href="#E298-MJMAIN-2212" x="3085" y="0"></use><use xlink:href="#E298-MJMATHI-71" x="4085" y="0"></use><use xlink:href="#E298-MJMAIN-28" x="4545" y="0"></use><g transform="translate(4934,0)"><use xlink:href="#E298-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E298-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E298-MJMAIN-2C" x="5902" y="0"></use><g transform="translate(6347,0)"><use xlink:href="#E298-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E298-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E298-MJMAIN-3B" x="7452" y="0"></use><use xlink:href="#E298-MJMAINB-77" x="7897" y="0"></use><use xlink:href="#E298-MJMAIN-29" x="8728" y="0"></use><g transform="translate(9117,0)"><use xlink:href="#E298-MJMAIN-5D" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E298-MJMAIN-32" x="393" y="583"></use></g></g></svg></span><script type="math/tex">\displaystyle \sum_{t=0}^{T-1}[G_t-q(S_t,A_t;\bold w)]^2</script><span> ，然后再沿着回合损失函数对 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="1.93ex" height="1.36ex" viewBox="0 -500.4 831 585.5" role="img" focusable="false" style="vertical-align: -0.198ex;"><defs><path stroke-width="0" id="E337-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E337-MJMAINB-77" x="0" y="0"></use></g></svg></span><script type="math/tex">\bold w</script><span> 的梯度反方向更新策略参数 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="1.93ex" height="1.36ex" viewBox="0 -500.4 831 585.5" role="img" focusable="false" style="vertical-align: -0.198ex;"><defs><path stroke-width="0" id="E337-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E337-MJMAINB-77" x="0" y="0"></use></g></svg></span><script type="math/tex">\bold w</script><span> 。</span></p><p><span>对于能够支持自动梯度计算的软件包，往往自带根据损失函数更新参数的功能。同样也可以自己计算梯度 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="13.622ex" height="2.71ex" viewBox="0 -832.7 5864.9 1166.9" role="img" focusable="false" style="vertical-align: -0.776ex;"><defs><path stroke-width="0" id="E301-MJMAIN-2207" d="M46 676Q46 679 51 683H781Q786 679 786 676Q786 674 617 326T444 -26Q439 -33 416 -33T388 -26Q385 -22 216 326T46 676ZM697 596Q697 597 445 597T193 596Q195 591 319 336T445 80L697 596Z"></path><path stroke-width="0" id="E301-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E301-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E301-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E301-MJMATHI-74" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path><path stroke-width="0" id="E301-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E301-MJMATHI-41" d="M208 74Q208 50 254 46Q272 46 272 35Q272 34 270 22Q267 8 264 4T251 0Q249 0 239 0T205 1T141 2Q70 2 50 0H42Q35 7 35 11Q37 38 48 46H62Q132 49 164 96Q170 102 345 401T523 704Q530 716 547 716H555H572Q578 707 578 706L606 383Q634 60 636 57Q641 46 701 46Q726 46 726 36Q726 34 723 22Q720 7 718 4T704 0Q701 0 690 0T651 1T578 2Q484 2 455 0H443Q437 6 437 9T439 27Q443 40 445 43L449 46H469Q523 49 533 63L521 213H283L249 155Q208 86 208 74ZM516 260Q516 271 504 416T490 562L463 519Q447 492 400 412L310 260L413 259Q516 259 516 260Z"></path><path stroke-width="0" id="E301-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E301-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E301-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E301-MJMAIN-2207" x="0" y="0"></use><use xlink:href="#E301-MJMATHI-71" x="833" y="0"></use><use xlink:href="#E301-MJMAIN-28" x="1293" y="0"></use><g transform="translate(1682,0)"><use xlink:href="#E301-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E301-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E301-MJMAIN-2C" x="2650" y="0"></use><g transform="translate(3094,0)"><use xlink:href="#E301-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E301-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E301-MJMAIN-3B" x="4200" y="0"></use><use xlink:href="#E301-MJMAINB-77" x="4644" y="0"></use><use xlink:href="#E301-MJMAIN-29" x="5475" y="0"></use></g></svg></span><script type="math/tex">\nabla q(S_t,A_t;\bold w)</script><span> ，然后利用下式更新：</span></p><div contenteditable="false" spellcheck="false" class="mathjax-block md-end-block md-math-block md-rawblock" id="mathjax-n7" cid="n7" mdtype="math_block"><div class="md-rawblock-container md-math-container" tabindex="-1"><div class="MathJax_SVG_Display"><span class="MathJax_SVG" id="MathJax-Element-237-Frame" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="98.296ex" height="5.025ex" viewBox="0 -1331 42321.7 2163.7" role="img" focusable="false" style="vertical-align: -1.775ex; margin-bottom: -0.159ex; max-width: 100%;"><defs><path stroke-width="0" id="E264-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E264-MJMAIN-31" d="M213 578L200 573Q186 568 160 563T102 556H83V602H102Q149 604 189 617T245 641T273 663Q275 666 285 666Q294 666 302 660V361L303 61Q310 54 315 52T339 48T401 46H427V0H416Q395 3 257 3Q121 3 100 0H88V46H114Q136 46 152 46T177 47T193 50T201 52T207 57T213 61V578Z"></path><path stroke-width="0" id="E264-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E264-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E264-MJMAIN-2190" d="M944 261T944 250T929 230H165Q167 228 182 216T211 189T244 152T277 96T303 25Q308 7 308 0Q308 -11 288 -11Q281 -11 278 -11T272 -7T267 2T263 21Q245 94 195 151T73 236Q58 242 55 247Q55 254 59 257T73 264Q121 283 158 314T215 375T247 434T264 480L267 497Q269 503 270 505T275 509T288 511Q308 511 308 500Q308 493 303 475Q293 438 278 406T246 352T215 315T185 287T165 270H929Q944 261 944 250Z"></path><path stroke-width="0" id="E264-MJMAIN-2212" d="M84 237T84 250T98 270H679Q694 262 694 250T679 230H98Q84 237 84 250Z"></path><path stroke-width="0" id="E264-MJMAIN-32" d="M109 429Q82 429 66 447T50 491Q50 562 103 614T235 666Q326 666 387 610T449 465Q449 422 429 383T381 315T301 241Q265 210 201 149L142 93L218 92Q375 92 385 97Q392 99 409 186V189H449V186Q448 183 436 95T421 3V0H50V19V31Q50 38 56 46T86 81Q115 113 136 137Q145 147 170 174T204 211T233 244T261 278T284 308T305 340T320 369T333 401T340 431T343 464Q343 527 309 573T212 619Q179 619 154 602T119 569T109 550Q109 549 114 549Q132 549 151 535T170 489Q170 464 154 447T109 429Z"></path><path stroke-width="0" id="E264-MJMATHI-3B1" d="M34 156Q34 270 120 356T309 442Q379 442 421 402T478 304Q484 275 485 237V208Q534 282 560 374Q564 388 566 390T582 393Q603 393 603 385Q603 376 594 346T558 261T497 161L486 147L487 123Q489 67 495 47T514 26Q528 28 540 37T557 60Q559 67 562 68T577 70Q597 70 597 62Q597 56 591 43Q579 19 556 5T512 -10H505Q438 -10 414 62L411 69L400 61Q390 53 370 41T325 18T267 -2T203 -11Q124 -11 79 39T34 156ZM208 26Q257 26 306 47T379 90L403 112Q401 255 396 290Q382 405 304 405Q235 405 183 332Q156 292 139 224T121 120Q121 71 146 49T208 26Z"></path><path stroke-width="0" id="E264-MJMATHI-74" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path><path stroke-width="0" id="E264-MJMAIN-2207" d="M46 676Q46 679 51 683H781Q786 679 786 676Q786 674 617 326T444 -26Q439 -33 416 -33T388 -26Q385 -22 216 326T46 676ZM697 596Q697 597 445 597T193 596Q195 591 319 336T445 80L697 596Z"></path><path stroke-width="0" id="E264-MJMAIN-5B" d="M118 -250V750H255V710H158V-210H255V-250H118Z"></path><path stroke-width="0" id="E264-MJMATHI-47" d="M50 252Q50 367 117 473T286 641T490 704Q580 704 633 653Q642 643 648 636T656 626L657 623Q660 623 684 649Q691 655 699 663T715 679T725 690L740 705H746Q760 705 760 698Q760 694 728 561Q692 422 692 421Q690 416 687 415T669 413H653Q647 419 647 422Q647 423 648 429T650 449T651 481Q651 552 619 605T510 659Q492 659 471 656T418 643T357 615T294 567T236 496T189 394T158 260Q156 242 156 221Q156 173 170 136T206 79T256 45T308 28T353 24Q407 24 452 47T514 106Q517 114 529 161T541 214Q541 222 528 224T468 227H431Q425 233 425 235T427 254Q431 267 437 273H454Q494 271 594 271Q634 271 659 271T695 272T707 272Q721 272 721 263Q721 261 719 249Q714 230 709 228Q706 227 694 227Q674 227 653 224Q646 221 643 215T629 164Q620 131 614 108Q589 6 586 3Q584 1 581 1Q571 1 553 21T530 52Q530 53 528 52T522 47Q448 -22 322 -22Q201 -22 126 55T50 252Z"></path><path stroke-width="0" id="E264-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E264-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E264-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E264-MJMATHI-41" d="M208 74Q208 50 254 46Q272 46 272 35Q272 34 270 22Q267 8 264 4T251 0Q249 0 239 0T205 1T141 2Q70 2 50 0H42Q35 7 35 11Q37 38 48 46H62Q132 49 164 96Q170 102 345 401T523 704Q530 716 547 716H555H572Q578 707 578 706L606 383Q634 60 636 57Q641 46 701 46Q726 46 726 36Q726 34 723 22Q720 7 718 4T704 0Q701 0 690 0T651 1T578 2Q484 2 455 0H443Q437 6 437 9T439 27Q443 40 445 43L449 46H469Q523 49 533 63L521 213H283L249 155Q208 86 208 74ZM516 260Q516 271 504 416T490 562L463 519Q447 492 400 412L310 260L413 259Q516 259 516 260Z"></path><path stroke-width="0" id="E264-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E264-MJMAIN-5D" d="M22 710V750H159V-250H22V-210H119V710H22Z"></path><path stroke-width="0" id="E264-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E264-MJMAIN-2B" d="M56 237T56 250T70 270H369V420L370 570Q380 583 389 583Q402 583 409 568V270H707Q722 262 722 250T707 230H409V-68Q401 -82 391 -82H389H387Q375 -82 369 -68V230H70Q56 237 56 250Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><g transform="translate(41043,0)"><g id="mjx-eqn-eq:1_1" transform="translate(0,-79)"><use xlink:href="#E264-MJMAIN-28"></use><use xlink:href="#E264-MJMAIN-31" x="389" y="0"></use><use xlink:href="#E264-MJMAIN-29" x="889" y="0"></use></g></g><g transform="translate(4470,0)"><g transform="translate(-19,0)"><g transform="translate(0,-79)"><use xlink:href="#E264-MJMAINB-77" x="0" y="0"></use><use xlink:href="#E264-MJMAIN-2190" x="1108" y="0"></use><use xlink:href="#E264-MJMAINB-77" x="2386" y="0"></use><use xlink:href="#E264-MJMAIN-2212" x="3439" y="0"></use><g transform="translate(4217,0)"><g transform="translate(342,0)"><rect stroke="none" width="620" height="60" x="0" y="220"></rect><use xlink:href="#E264-MJMAIN-31" x="60" y="676"></use><use xlink:href="#E264-MJMAIN-32" x="60" y="-686"></use></g></g><g transform="translate(5300,0)"><use xlink:href="#E264-MJMATHI-3B1" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E264-MJMATHI-74" x="905" y="-213"></use></g><use xlink:href="#E264-MJMAIN-2207" x="6295" y="0"></use><use xlink:href="#E264-MJMAIN-5B" x="7128" y="0"></use><g transform="translate(7406,0)"><use xlink:href="#E264-MJMATHI-47" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E264-MJMATHI-74" x="1111" y="-213"></use></g><use xlink:href="#E264-MJMAIN-2212" x="8769" y="0"></use><use xlink:href="#E264-MJMATHI-71" x="9769" y="0"></use><use xlink:href="#E264-MJMAIN-28" x="10229" y="0"></use><g transform="translate(10618,0)"><use xlink:href="#E264-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E264-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E264-MJMAIN-2C" x="11587" y="0"></use><g transform="translate(12031,0)"><use xlink:href="#E264-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E264-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E264-MJMAIN-3B" x="13137" y="0"></use><use xlink:href="#E264-MJMAINB-77" x="13581" y="0"></use><use xlink:href="#E264-MJMAIN-29" x="14412" y="0"></use><g transform="translate(14801,0)"><use xlink:href="#E264-MJMAIN-5D" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E264-MJMAIN-32" x="393" y="583"></use></g><use xlink:href="#E264-MJMAIN-3D" x="15811" y="0"></use><use xlink:href="#E264-MJMAINB-77" x="16866" y="0"></use><use xlink:href="#E264-MJMAIN-2B" x="17920" y="0"></use><g transform="translate(18920,0)"><use xlink:href="#E264-MJMATHI-3B1" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E264-MJMATHI-74" x="905" y="-213"></use></g><use xlink:href="#E264-MJMAIN-5B" x="19915" y="0"></use><g transform="translate(20193,0)"><use xlink:href="#E264-MJMATHI-47" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E264-MJMATHI-74" x="1111" y="-213"></use></g><use xlink:href="#E264-MJMAIN-2212" x="21557" y="0"></use><use xlink:href="#E264-MJMATHI-71" x="22557" y="0"></use><use xlink:href="#E264-MJMAIN-28" x="23017" y="0"></use><g transform="translate(23406,0)"><use xlink:href="#E264-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E264-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E264-MJMAIN-2C" x="24374" y="0"></use><g transform="translate(24819,0)"><use xlink:href="#E264-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E264-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E264-MJMAIN-3B" x="25924" y="0"></use><use xlink:href="#E264-MJMAINB-77" x="26369" y="0"></use><use xlink:href="#E264-MJMAIN-29" x="27200" y="0"></use><use xlink:href="#E264-MJMAIN-5D" x="27589" y="0"></use><use xlink:href="#E264-MJMAIN-2207" x="27867" y="0"></use><use xlink:href="#E264-MJMATHI-71" x="28700" y="0"></use><use xlink:href="#E264-MJMAIN-28" x="29160" y="0"></use><g transform="translate(29549,0)"><use xlink:href="#E264-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E264-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E264-MJMAIN-2C" x="30517" y="0"></use><g transform="translate(30962,0)"><use xlink:href="#E264-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E264-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E264-MJMAIN-3B" x="32067" y="0"></use><use xlink:href="#E264-MJMAINB-77" x="32512" y="0"></use><use xlink:href="#E264-MJMAIN-29" x="33343" y="0"></use></g></g></g></g></svg></span></div><script type="math/tex; mode=display" id="MathJax-Element-237">\bold w \leftarrow \bold w - \frac{1}{2} \alpha_t \nabla[G_t - q(S_t,A_t;\bold w)]^2 = \bold w + \alpha_t [G_t - q(S_t,A_t;\bold w)] \nabla q(S_t,A_t;\bold w)
\label{eq:1}</script></div></div><p><span>对状态价值函数也可以类似的定义回合损失函数 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="19.334ex" height="7.051ex" viewBox="0 -1787.9 8324.2 3035.9" role="img" focusable="false" style="vertical-align: -2.899ex;"><defs><path stroke-width="0" id="E302-MJSZ2-2211" d="M60 948Q63 950 665 950H1267L1325 815Q1384 677 1388 669H1348L1341 683Q1320 724 1285 761Q1235 809 1174 838T1033 881T882 898T699 902H574H543H251L259 891Q722 258 724 252Q725 250 724 246Q721 243 460 -56L196 -356Q196 -357 407 -357Q459 -357 548 -357T676 -358Q812 -358 896 -353T1063 -332T1204 -283T1307 -196Q1328 -170 1348 -124H1388Q1388 -125 1381 -145T1356 -210T1325 -294L1267 -449L666 -450Q64 -450 61 -448Q55 -446 55 -439Q55 -437 57 -433L590 177Q590 178 557 222T452 366T322 544L56 909L55 924Q55 945 60 948Z"></path><path stroke-width="0" id="E302-MJMATHI-74" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path><path stroke-width="0" id="E302-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E302-MJMAIN-30" d="M96 585Q152 666 249 666Q297 666 345 640T423 548Q460 465 460 320Q460 165 417 83Q397 41 362 16T301 -15T250 -22Q224 -22 198 -16T137 16T82 83Q39 165 39 320Q39 494 96 585ZM321 597Q291 629 250 629Q208 629 178 597Q153 571 145 525T137 333Q137 175 145 125T181 46Q209 16 250 16Q290 16 318 46Q347 76 354 130T362 333Q362 478 354 524T321 597Z"></path><path stroke-width="0" id="E302-MJMATHI-54" d="M40 437Q21 437 21 445Q21 450 37 501T71 602L88 651Q93 669 101 677H569H659Q691 677 697 676T704 667Q704 661 687 553T668 444Q668 437 649 437Q640 437 637 437T631 442L629 445Q629 451 635 490T641 551Q641 586 628 604T573 629Q568 630 515 631Q469 631 457 630T439 622Q438 621 368 343T298 60Q298 48 386 46Q418 46 427 45T436 36Q436 31 433 22Q429 4 424 1L422 0Q419 0 415 0Q410 0 363 1T228 2Q99 2 64 0H49Q43 6 43 9T45 27Q49 40 55 46H83H94Q174 46 189 55Q190 56 191 56Q196 59 201 76T241 233Q258 301 269 344Q339 619 339 625Q339 630 310 630H279Q212 630 191 624Q146 614 121 583T67 467Q60 445 57 441T43 437H40Z"></path><path stroke-width="0" id="E302-MJMAIN-5B" d="M118 -250V750H255V710H158V-210H255V-250H118Z"></path><path stroke-width="0" id="E302-MJMATHI-47" d="M50 252Q50 367 117 473T286 641T490 704Q580 704 633 653Q642 643 648 636T656 626L657 623Q660 623 684 649Q691 655 699 663T715 679T725 690L740 705H746Q760 705 760 698Q760 694 728 561Q692 422 692 421Q690 416 687 415T669 413H653Q647 419 647 422Q647 423 648 429T650 449T651 481Q651 552 619 605T510 659Q492 659 471 656T418 643T357 615T294 567T236 496T189 394T158 260Q156 242 156 221Q156 173 170 136T206 79T256 45T308 28T353 24Q407 24 452 47T514 106Q517 114 529 161T541 214Q541 222 528 224T468 227H431Q425 233 425 235T427 254Q431 267 437 273H454Q494 271 594 271Q634 271 659 271T695 272T707 272Q721 272 721 263Q721 261 719 249Q714 230 709 228Q706 227 694 227Q674 227 653 224Q646 221 643 215T629 164Q620 131 614 108Q589 6 586 3Q584 1 581 1Q571 1 553 21T530 52Q530 53 528 52T522 47Q448 -22 322 -22Q201 -22 126 55T50 252Z"></path><path stroke-width="0" id="E302-MJMAIN-2212" d="M84 237T84 250T98 270H679Q694 262 694 250T679 230H98Q84 237 84 250Z"></path><path stroke-width="0" id="E302-MJMATHI-76" d="M173 380Q173 405 154 405Q130 405 104 376T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Q21 294 29 316T53 368T97 419T160 441Q202 441 225 417T249 361Q249 344 246 335Q246 329 231 291T200 202T182 113Q182 86 187 69Q200 26 250 26Q287 26 319 60T369 139T398 222T409 277Q409 300 401 317T383 343T365 361T357 383Q357 405 376 424T417 443Q436 443 451 425T467 367Q467 340 455 284T418 159T347 40T241 -11Q177 -11 139 22Q102 54 102 117Q102 148 110 181T151 298Q173 362 173 380Z"></path><path stroke-width="0" id="E302-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E302-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E302-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E302-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E302-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E302-MJMAIN-5D" d="M22 710V750H159V-250H22V-210H119V710H22Z"></path><path stroke-width="0" id="E302-MJMAIN-32" d="M109 429Q82 429 66 447T50 491Q50 562 103 614T235 666Q326 666 387 610T449 465Q449 422 429 383T381 315T301 241Q265 210 201 149L142 93L218 92Q375 92 385 97Q392 99 409 186V189H449V186Q448 183 436 95T421 3V0H50V19V31Q50 38 56 46T86 81Q115 113 136 137Q145 147 170 174T204 211T233 244T261 278T284 308T305 340T320 369T333 401T340 431T343 464Q343 527 309 573T212 619Q179 619 154 602T119 569T109 550Q109 549 114 549Q132 549 151 535T170 489Q170 464 154 447T109 429Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E302-MJSZ2-2211" x="0" y="0"></use><g transform="translate(142,-1088)"><use transform="scale(0.707)" xlink:href="#E302-MJMATHI-74" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E302-MJMAIN-3D" x="361" y="0"></use><use transform="scale(0.707)" xlink:href="#E302-MJMAIN-30" x="1139" y="0"></use></g><use transform="scale(0.707)" xlink:href="#E302-MJMATHI-54" x="669" y="1626"></use><use xlink:href="#E302-MJMAIN-5B" x="1444" y="0"></use><g transform="translate(1722,0)"><use xlink:href="#E302-MJMATHI-47" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E302-MJMATHI-74" x="1111" y="-213"></use></g><use xlink:href="#E302-MJMAIN-2212" x="3085" y="0"></use><use xlink:href="#E302-MJMATHI-76" x="4085" y="0"></use><use xlink:href="#E302-MJMAIN-28" x="4570" y="0"></use><g transform="translate(4959,0)"><use xlink:href="#E302-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E302-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E302-MJMAIN-3B" x="5927" y="0"></use><use xlink:href="#E302-MJMAINB-77" x="6372" y="0"></use><use xlink:href="#E302-MJMAIN-29" x="7203" y="0"></use><g transform="translate(7592,0)"><use xlink:href="#E302-MJMAIN-5D" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E302-MJMAIN-32" x="393" y="583"></use></g></g></svg></span><script type="math/tex">\displaystyle \sum_{t=0}^{T}[G_t-v(S_t;\bold w)]^2</script><span> ，其对应的更新式为：</span></p><div contenteditable="false" spellcheck="false" class="mathjax-block md-end-block md-math-block md-rawblock" id="mathjax-n9" cid="n9" mdtype="math_block"><div class="md-rawblock-container md-math-container" tabindex="-1"><div class="MathJax_SVG_Display"><span class="MathJax_SVG" id="MathJax-Element-238-Frame" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="98.296ex" height="5.025ex" viewBox="0 -1331 42321.7 2163.7" role="img" focusable="false" style="vertical-align: -1.775ex; margin-bottom: -0.159ex; max-width: 100%;"><defs><path stroke-width="0" id="E265-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E265-MJMAIN-32" d="M109 429Q82 429 66 447T50 491Q50 562 103 614T235 666Q326 666 387 610T449 465Q449 422 429 383T381 315T301 241Q265 210 201 149L142 93L218 92Q375 92 385 97Q392 99 409 186V189H449V186Q448 183 436 95T421 3V0H50V19V31Q50 38 56 46T86 81Q115 113 136 137Q145 147 170 174T204 211T233 244T261 278T284 308T305 340T320 369T333 401T340 431T343 464Q343 527 309 573T212 619Q179 619 154 602T119 569T109 550Q109 549 114 549Q132 549 151 535T170 489Q170 464 154 447T109 429Z"></path><path stroke-width="0" id="E265-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E265-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E265-MJMAIN-2190" d="M944 261T944 250T929 230H165Q167 228 182 216T211 189T244 152T277 96T303 25Q308 7 308 0Q308 -11 288 -11Q281 -11 278 -11T272 -7T267 2T263 21Q245 94 195 151T73 236Q58 242 55 247Q55 254 59 257T73 264Q121 283 158 314T215 375T247 434T264 480L267 497Q269 503 270 505T275 509T288 511Q308 511 308 500Q308 493 303 475Q293 438 278 406T246 352T215 315T185 287T165 270H929Q944 261 944 250Z"></path><path stroke-width="0" id="E265-MJMAIN-2212" d="M84 237T84 250T98 270H679Q694 262 694 250T679 230H98Q84 237 84 250Z"></path><path stroke-width="0" id="E265-MJMAIN-31" d="M213 578L200 573Q186 568 160 563T102 556H83V602H102Q149 604 189 617T245 641T273 663Q275 666 285 666Q294 666 302 660V361L303 61Q310 54 315 52T339 48T401 46H427V0H416Q395 3 257 3Q121 3 100 0H88V46H114Q136 46 152 46T177 47T193 50T201 52T207 57T213 61V578Z"></path><path stroke-width="0" id="E265-MJMATHI-3B1" d="M34 156Q34 270 120 356T309 442Q379 442 421 402T478 304Q484 275 485 237V208Q534 282 560 374Q564 388 566 390T582 393Q603 393 603 385Q603 376 594 346T558 261T497 161L486 147L487 123Q489 67 495 47T514 26Q528 28 540 37T557 60Q559 67 562 68T577 70Q597 70 597 62Q597 56 591 43Q579 19 556 5T512 -10H505Q438 -10 414 62L411 69L400 61Q390 53 370 41T325 18T267 -2T203 -11Q124 -11 79 39T34 156ZM208 26Q257 26 306 47T379 90L403 112Q401 255 396 290Q382 405 304 405Q235 405 183 332Q156 292 139 224T121 120Q121 71 146 49T208 26Z"></path><path stroke-width="0" id="E265-MJMATHI-74" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path><path stroke-width="0" id="E265-MJMAIN-2207" d="M46 676Q46 679 51 683H781Q786 679 786 676Q786 674 617 326T444 -26Q439 -33 416 -33T388 -26Q385 -22 216 326T46 676ZM697 596Q697 597 445 597T193 596Q195 591 319 336T445 80L697 596Z"></path><path stroke-width="0" id="E265-MJMAIN-5B" d="M118 -250V750H255V710H158V-210H255V-250H118Z"></path><path stroke-width="0" id="E265-MJMATHI-47" d="M50 252Q50 367 117 473T286 641T490 704Q580 704 633 653Q642 643 648 636T656 626L657 623Q660 623 684 649Q691 655 699 663T715 679T725 690L740 705H746Q760 705 760 698Q760 694 728 561Q692 422 692 421Q690 416 687 415T669 413H653Q647 419 647 422Q647 423 648 429T650 449T651 481Q651 552 619 605T510 659Q492 659 471 656T418 643T357 615T294 567T236 496T189 394T158 260Q156 242 156 221Q156 173 170 136T206 79T256 45T308 28T353 24Q407 24 452 47T514 106Q517 114 529 161T541 214Q541 222 528 224T468 227H431Q425 233 425 235T427 254Q431 267 437 273H454Q494 271 594 271Q634 271 659 271T695 272T707 272Q721 272 721 263Q721 261 719 249Q714 230 709 228Q706 227 694 227Q674 227 653 224Q646 221 643 215T629 164Q620 131 614 108Q589 6 586 3Q584 1 581 1Q571 1 553 21T530 52Q530 53 528 52T522 47Q448 -22 322 -22Q201 -22 126 55T50 252Z"></path><path stroke-width="0" id="E265-MJMATHI-76" d="M173 380Q173 405 154 405Q130 405 104 376T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Q21 294 29 316T53 368T97 419T160 441Q202 441 225 417T249 361Q249 344 246 335Q246 329 231 291T200 202T182 113Q182 86 187 69Q200 26 250 26Q287 26 319 60T369 139T398 222T409 277Q409 300 401 317T383 343T365 361T357 383Q357 405 376 424T417 443Q436 443 451 425T467 367Q467 340 455 284T418 159T347 40T241 -11Q177 -11 139 22Q102 54 102 117Q102 148 110 181T151 298Q173 362 173 380Z"></path><path stroke-width="0" id="E265-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E265-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E265-MJMAIN-5D" d="M22 710V750H159V-250H22V-210H119V710H22Z"></path><path stroke-width="0" id="E265-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E265-MJMAIN-2B" d="M56 237T56 250T70 270H369V420L370 570Q380 583 389 583Q402 583 409 568V270H707Q722 262 722 250T707 230H409V-68Q401 -82 391 -82H389H387Q375 -82 369 -68V230H70Q56 237 56 250Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><g transform="translate(41043,0)"><g id="mjx-eqn-eq:2" transform="translate(0,-79)"><use xlink:href="#E265-MJMAIN-28"></use><use xlink:href="#E265-MJMAIN-32" x="389" y="0"></use><use xlink:href="#E265-MJMAIN-29" x="889" y="0"></use></g></g><g transform="translate(6758,0)"><g transform="translate(-19,0)"><g transform="translate(0,-79)"><use xlink:href="#E265-MJMAINB-77" x="0" y="0"></use><use xlink:href="#E265-MJMAIN-2190" x="1108" y="0"></use><use xlink:href="#E265-MJMAINB-77" x="2386" y="0"></use><use xlink:href="#E265-MJMAIN-2212" x="3439" y="0"></use><g transform="translate(4217,0)"><g transform="translate(342,0)"><rect stroke="none" width="620" height="60" x="0" y="220"></rect><use xlink:href="#E265-MJMAIN-31" x="60" y="676"></use><use xlink:href="#E265-MJMAIN-32" x="60" y="-686"></use></g></g><g transform="translate(5300,0)"><use xlink:href="#E265-MJMATHI-3B1" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E265-MJMATHI-74" x="905" y="-213"></use></g><use xlink:href="#E265-MJMAIN-2207" x="6295" y="0"></use><use xlink:href="#E265-MJMAIN-5B" x="7128" y="0"></use><g transform="translate(7406,0)"><use xlink:href="#E265-MJMATHI-47" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E265-MJMATHI-74" x="1111" y="-213"></use></g><use xlink:href="#E265-MJMAIN-2212" x="8769" y="0"></use><use xlink:href="#E265-MJMATHI-76" x="9769" y="0"></use><use xlink:href="#E265-MJMAIN-28" x="10254" y="0"></use><g transform="translate(10643,0)"><use xlink:href="#E265-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E265-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E265-MJMAIN-3B" x="11612" y="0"></use><use xlink:href="#E265-MJMAINB-77" x="12056" y="0"></use><use xlink:href="#E265-MJMAIN-29" x="12887" y="0"></use><g transform="translate(13276,0)"><use xlink:href="#E265-MJMAIN-5D" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E265-MJMAIN-32" x="393" y="583"></use></g><use xlink:href="#E265-MJMAIN-3D" x="14286" y="0"></use><use xlink:href="#E265-MJMAINB-77" x="15342" y="0"></use><use xlink:href="#E265-MJMAIN-2B" x="16395" y="0"></use><g transform="translate(17395,0)"><use xlink:href="#E265-MJMATHI-3B1" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E265-MJMATHI-74" x="905" y="-213"></use></g><use xlink:href="#E265-MJMAIN-5B" x="18390" y="0"></use><g transform="translate(18668,0)"><use xlink:href="#E265-MJMATHI-47" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E265-MJMATHI-74" x="1111" y="-213"></use></g><use xlink:href="#E265-MJMAIN-2212" x="20032" y="0"></use><use xlink:href="#E265-MJMATHI-76" x="21032" y="0"></use><use xlink:href="#E265-MJMAIN-28" x="21517" y="0"></use><g transform="translate(21906,0)"><use xlink:href="#E265-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E265-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E265-MJMAIN-3B" x="22874" y="0"></use><use xlink:href="#E265-MJMAINB-77" x="23319" y="0"></use><use xlink:href="#E265-MJMAIN-29" x="24150" y="0"></use><use xlink:href="#E265-MJMAIN-5D" x="24539" y="0"></use><use xlink:href="#E265-MJMAIN-2207" x="24817" y="0"></use><use xlink:href="#E265-MJMATHI-76" x="25650" y="0"></use><use xlink:href="#E265-MJMAIN-28" x="26135" y="0"></use><g transform="translate(26524,0)"><use xlink:href="#E265-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E265-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E265-MJMAIN-3B" x="27492" y="0"></use><use xlink:href="#E265-MJMAINB-77" x="27937" y="0"></use><use xlink:href="#E265-MJMAIN-29" x="28768" y="0"></use></g></g></g></g></svg></span></div><script type="math/tex; mode=display" id="MathJax-Element-238">\bold w \leftarrow \bold w - \frac{1}{2} \alpha_t \nabla[G_t - v(S_t;\bold w)]^2 = \bold w + \alpha_t [G_t - v(S_t;\bold w)] \nabla v(S_t;\bold w)
\label{eq:2}</script></div></div><p><span>将同策回合更新价值估计与函数近似法相结合，并在更新价值函数时使用随机梯度下降算法，就能得到算法 6-1 ：</span></p><div contenteditable="false" spellcheck="false" class="mathjax-block md-end-block md-math-block md-rawblock" id="mathjax-n11" cid="n11" mdtype="math_block"><div class="md-rawblock-container md-math-container" tabindex="-1"><div class="MathJax_SVG_Display"><span class="MathJax_SVG" id="MathJax-Element-239-Frame" tabindex="-1" style="font-size: 100%; display: inline-block; zoom: 0.971345;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="101.294ex" height="38.691ex" viewBox="-18.1 -43.5 43612.5 16658.5" role="img" focusable="false" style="vertical-align: -38.59ex; margin-left: -0.042ex; max-width: 100%;"><defs><path stroke-width="0" id="E266-MJMAINB-36" d="M48 318Q48 395 68 456T120 553T193 613T273 646T350 655Q425 655 461 616T497 524Q497 485 475 468T428 451Q399 451 378 470T357 521Q357 565 403 588Q375 601 351 601Q313 601 282 584Q242 565 222 526Q199 473 199 367Q201 369 210 380T227 396T246 410T275 422T312 426Q438 426 494 332Q526 285 526 208V199Q526 112 465 53Q428 17 388 3T285 -11Q236 -11 195 7T135 43T104 80Q48 165 48 318ZM375 231V244V268Q375 295 373 310T364 342T341 366T299 374H297Q231 374 208 287Q200 257 200 196Q201 120 209 100Q231 47 288 47Q351 47 368 90Q375 112 375 231Z"></path><path stroke-width="0" id="E266-MJMAINB-2D" d="M13 166V278H318V166H13Z"></path><path stroke-width="0" id="E266-MJMAINB-31" d="M481 0L294 3Q136 3 109 0H96V62H227V304Q227 546 225 546Q169 529 97 529H80V591H97Q231 591 308 647L319 655H333Q355 655 359 644Q361 640 361 351V62H494V0H481Z"></path><path stroke-width="0" id="E266-MJMAIN-31" d="M213 578L200 573Q186 568 160 563T102 556H83V602H102Q149 604 189 617T245 641T273 663Q275 666 285 666Q294 666 302 660V361L303 61Q310 54 315 52T339 48T401 46H427V0H416Q395 3 257 3Q121 3 100 0H88V46H114Q136 46 152 46T177 47T193 50T201 52T207 57T213 61V578Z"></path><path stroke-width="0" id="E266-MJMAIN-2E" d="M78 60Q78 84 95 102T138 120Q162 120 180 104T199 61Q199 36 182 18T139 0T96 17T78 60Z"></path><path stroke-width="0" id="E266-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E266-MJMAIN-32" d="M109 429Q82 429 66 447T50 491Q50 562 103 614T235 666Q326 666 387 610T449 465Q449 422 429 383T381 315T301 241Q265 210 201 149L142 93L218 92Q375 92 385 97Q392 99 409 186V189H449V186Q448 183 436 95T421 3V0H50V19V31Q50 38 56 46T86 81Q115 113 136 137Q145 147 170 174T204 211T233 244T261 278T284 308T305 340T320 369T333 401T340 431T343 464Q343 527 309 573T212 619Q179 619 154 602T119 569T109 550Q109 549 114 549Q132 549 151 535T170 489Q170 464 154 447T109 429Z"></path><path stroke-width="0" id="E266-MJMATHI-3C0" d="M132 -11Q98 -11 98 22V33L111 61Q186 219 220 334L228 358H196Q158 358 142 355T103 336Q92 329 81 318T62 297T53 285Q51 284 38 284Q19 284 19 294Q19 300 38 329T93 391T164 429Q171 431 389 431Q549 431 553 430Q573 423 573 402Q573 371 541 360Q535 358 472 358H408L405 341Q393 269 393 222Q393 170 402 129T421 65T431 37Q431 20 417 5T381 -10Q370 -10 363 -7T347 17T331 77Q330 86 330 121Q330 170 339 226T357 318T367 358H269L268 354Q268 351 249 275T206 114T175 17Q164 -11 132 -11Z"></path><path stroke-width="0" id="E266-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E266-MJMAIN-30" d="M96 585Q152 666 249 666Q297 666 345 640T423 548Q460 465 460 320Q460 165 417 83Q397 41 362 16T301 -15T250 -22Q224 -22 198 -16T137 16T82 83Q39 165 39 320Q39 494 96 585ZM321 597Q291 629 250 629Q208 629 178 597Q153 571 145 525T137 333Q137 175 145 125T181 46Q209 16 250 16Q290 16 318 46Q347 76 354 130T362 333Q362 478 354 524T321 597Z"></path><path stroke-width="0" id="E266-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E266-MJMATHI-41" d="M208 74Q208 50 254 46Q272 46 272 35Q272 34 270 22Q267 8 264 4T251 0Q249 0 239 0T205 1T141 2Q70 2 50 0H42Q35 7 35 11Q37 38 48 46H62Q132 49 164 96Q170 102 345 401T523 704Q530 716 547 716H555H572Q578 707 578 706L606 383Q634 60 636 57Q641 46 701 46Q726 46 726 36Q726 34 723 22Q720 7 718 4T704 0Q701 0 690 0T651 1T578 2Q484 2 455 0H443Q437 6 437 9T439 27Q443 40 445 43L449 46H469Q523 49 533 63L521 213H283L249 155Q208 86 208 74ZM516 260Q516 271 504 416T490 562L463 519Q447 492 400 412L310 260L413 259Q516 259 516 260Z"></path><path stroke-width="0" id="E266-MJMATHI-52" d="M230 637Q203 637 198 638T193 649Q193 676 204 682Q206 683 378 683Q550 682 564 680Q620 672 658 652T712 606T733 563T739 529Q739 484 710 445T643 385T576 351T538 338L545 333Q612 295 612 223Q612 212 607 162T602 80V71Q602 53 603 43T614 25T640 16Q668 16 686 38T712 85Q717 99 720 102T735 105Q755 105 755 93Q755 75 731 36Q693 -21 641 -21H632Q571 -21 531 4T487 82Q487 109 502 166T517 239Q517 290 474 313Q459 320 449 321T378 323H309L277 193Q244 61 244 59Q244 55 245 54T252 50T269 48T302 46H333Q339 38 339 37T336 19Q332 6 326 0H311Q275 2 180 2Q146 2 117 2T71 2T50 1Q33 1 33 10Q33 12 36 24Q41 43 46 45Q50 46 61 46H67Q94 46 127 49Q141 52 146 61Q149 65 218 339T287 628Q287 635 230 637ZM630 554Q630 586 609 608T523 636Q521 636 500 636T462 637H440Q393 637 386 627Q385 624 352 494T319 361Q319 360 388 360Q466 361 492 367Q556 377 592 426Q608 449 619 486T630 554Z"></path><path stroke-width="0" id="E266-MJMAIN-22EF" d="M78 250Q78 274 95 292T138 310Q162 310 180 294T199 251Q199 226 182 208T139 190T96 207T78 250ZM525 250Q525 274 542 292T585 310Q609 310 627 294T646 251Q646 226 629 208T586 190T543 207T525 250ZM972 250Q972 274 989 292T1032 310Q1056 310 1074 294T1093 251Q1093 226 1076 208T1033 190T990 207T972 250Z"></path><path stroke-width="0" id="E266-MJMATHI-54" d="M40 437Q21 437 21 445Q21 450 37 501T71 602L88 651Q93 669 101 677H569H659Q691 677 697 676T704 667Q704 661 687 553T668 444Q668 437 649 437Q640 437 637 437T631 442L629 445Q629 451 635 490T641 551Q641 586 628 604T573 629Q568 630 515 631Q469 631 457 630T439 622Q438 621 368 343T298 60Q298 48 386 46Q418 46 427 45T436 36Q436 31 433 22Q429 4 424 1L422 0Q419 0 415 0Q410 0 363 1T228 2Q99 2 64 0H49Q43 6 43 9T45 27Q49 40 55 46H83H94Q174 46 189 55Q190 56 191 56Q196 59 201 76T241 233Q258 301 269 344Q339 619 339 625Q339 630 310 630H279Q212 630 191 624Q146 614 121 583T67 467Q60 445 57 441T43 437H40Z"></path><path stroke-width="0" id="E266-MJMAIN-2212" d="M84 237T84 250T98 270H679Q694 262 694 250T679 230H98Q84 237 84 250Z"></path><path stroke-width="0" id="E266-MJMATHI-47" d="M50 252Q50 367 117 473T286 641T490 704Q580 704 633 653Q642 643 648 636T656 626L657 623Q660 623 684 649Q691 655 699 663T715 679T725 690L740 705H746Q760 705 760 698Q760 694 728 561Q692 422 692 421Q690 416 687 415T669 413H653Q647 419 647 422Q647 423 648 429T650 449T651 481Q651 552 619 605T510 659Q492 659 471 656T418 643T357 615T294 567T236 496T189 394T158 260Q156 242 156 221Q156 173 170 136T206 79T256 45T308 28T353 24Q407 24 452 47T514 106Q517 114 529 161T541 214Q541 222 528 224T468 227H431Q425 233 425 235T427 254Q431 267 437 273H454Q494 271 594 271Q634 271 659 271T695 272T707 272Q721 272 721 263Q721 261 719 249Q714 230 709 228Q706 227 694 227Q674 227 653 224Q646 221 643 215T629 164Q620 131 614 108Q589 6 586 3Q584 1 581 1Q571 1 553 21T530 52Q530 53 528 52T522 47Q448 -22 322 -22Q201 -22 126 55T50 252Z"></path><path stroke-width="0" id="E266-MJMAIN-2190" d="M944 261T944 250T929 230H165Q167 228 182 216T211 189T244 152T277 96T303 25Q308 7 308 0Q308 -11 288 -11Q281 -11 278 -11T272 -7T267 2T263 21Q245 94 195 151T73 236Q58 242 55 247Q55 254 59 257T73 264Q121 283 158 314T215 375T247 434T264 480L267 497Q269 503 270 505T275 509T288 511Q308 511 308 500Q308 493 303 475Q293 438 278 406T246 352T215 315T185 287T165 270H929Q944 261 944 250Z"></path><path stroke-width="0" id="E266-MJMAIN-33" d="M127 463Q100 463 85 480T69 524Q69 579 117 622T233 665Q268 665 277 664Q351 652 390 611T430 522Q430 470 396 421T302 350L299 348Q299 347 308 345T337 336T375 315Q457 262 457 175Q457 96 395 37T238 -22Q158 -22 100 21T42 130Q42 158 60 175T105 193Q133 193 151 175T169 130Q169 119 166 110T159 94T148 82T136 74T126 70T118 67L114 66Q165 21 238 21Q293 21 321 74Q338 107 338 175V195Q338 290 274 322Q259 328 213 329L171 330L168 332Q166 335 166 348Q166 366 174 366Q202 366 232 371Q266 376 294 413T322 525V533Q322 590 287 612Q265 626 240 626Q208 626 181 615T143 592T132 580H135Q138 579 143 578T153 573T165 566T175 555T183 540T186 520Q186 498 172 481T127 463Z"></path><path stroke-width="0" id="E266-MJMATHI-74" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path><path stroke-width="0" id="E266-MJMATHI-3B3" d="M31 249Q11 249 11 258Q11 275 26 304T66 365T129 418T206 441Q233 441 239 440Q287 429 318 386T371 255Q385 195 385 170Q385 166 386 166L398 193Q418 244 443 300T486 391T508 430Q510 431 524 431H537Q543 425 543 422Q543 418 522 378T463 251T391 71Q385 55 378 6T357 -100Q341 -165 330 -190T303 -216Q286 -216 286 -188Q286 -138 340 32L346 51L347 69Q348 79 348 100Q348 257 291 317Q251 355 196 355Q148 355 108 329T51 260Q49 251 47 251Q45 249 31 249Z"></path><path stroke-width="0" id="E266-MJMAIN-2B" d="M56 237T56 250T70 270H369V420L370 570Q380 583 389 583Q402 583 409 568V270H707Q722 262 722 250T707 230H409V-68Q401 -82 391 -82H389H387Q375 -82 369 -68V230H70Q56 237 56 250Z"></path><path stroke-width="0" id="E266-MJMAIN-5B" d="M118 -250V750H255V710H158V-210H255V-250H118Z"></path><path stroke-width="0" id="E266-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E266-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E266-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E266-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E266-MJMAIN-5D" d="M22 710V750H159V-250H22V-210H119V710H22Z"></path><path stroke-width="0" id="E266-MJMATHI-76" d="M173 380Q173 405 154 405Q130 405 104 376T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Q21 294 29 316T53 368T97 419T160 441Q202 441 225 417T249 361Q249 344 246 335Q246 329 231 291T200 202T182 113Q182 86 187 69Q200 26 250 26Q287 26 319 60T369 139T398 222T409 277Q409 300 401 317T383 343T365 361T357 383Q357 405 376 424T417 443Q436 443 451 425T467 367Q467 340 455 284T418 159T347 40T241 -11Q177 -11 139 22Q102 54 102 117Q102 148 110 181T151 298Q173 362 173 380Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><g transform="translate(9775,-2462)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">算</text><g transform="translate(1052,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">法</text></g><use transform="scale(1.2)" xlink:href="#E266-MJMAINB-36" x="1963" y="0"></use><use transform="scale(1.2)" xlink:href="#E266-MJMAINB-2D" x="2538" y="0"></use><use transform="scale(1.2)" xlink:href="#E266-MJMAINB-31" x="2921" y="0"></use><g transform="translate(4945,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">随</text></g><g transform="translate(5998,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">机</text></g><g transform="translate(7050,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">梯</text></g><g transform="translate(8103,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">度</text></g><g transform="translate(9156,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">下</text></g><g transform="translate(10209,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">降</text></g><g transform="translate(11262,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">函</text></g><g transform="translate(12278,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">数</text></g><g transform="translate(13331,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">近</text></g><g transform="translate(14384,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">似</text></g><g transform="translate(15437,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">评</text></g><g transform="translate(16490,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">估</text></g><g transform="translate(17542,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">策</text></g><g transform="translate(18595,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">略</text></g><g transform="translate(19648,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">的</text></g><g transform="translate(20664,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">价</text></g><g transform="translate(21717,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">值</text></g></g><g transform="translate(0,-9238)"><g transform="translate(-19,0)"><g transform="translate(0,5378)"><g><rect fill="black" stroke="none" width="1569" height="100" x="0" y="500"></rect></g></g><g transform="translate(0,-5179)"><g><rect fill="black" stroke="none" width="1569" height="100" x="0" y="-500"></rect></g></g></g><g transform="translate(1551,0)"><g transform="translate(0,5378)"><g><rect fill="black" stroke="none" width="41597" height="100" x="0" y="500"></rect></g></g><g transform="translate(0,4078)"><use xlink:href="#E266-MJMAIN-31"></use><use xlink:href="#E266-MJMAIN-2E" x="500" y="0"></use><g transform="translate(778,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(1608,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">初</text></g><g transform="translate(2439,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">始</text></g><g transform="translate(3269,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">化</text></g><g transform="translate(4100,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(4931,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">任</text></g><g transform="translate(5761,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">意</text></g><g transform="translate(6592,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">初</text></g><g transform="translate(7423,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">始</text></g><g transform="translate(8253,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">化</text></g><g transform="translate(9084,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">参</text></g><g transform="translate(9915,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">数</text></g><use xlink:href="#E266-MJMAINB-77" x="10995" y="0"></use><g transform="translate(11826,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">。</text></g></g></g><g transform="translate(0,2778)"><use xlink:href="#E266-MJMAIN-32"></use><use xlink:href="#E266-MJMAIN-2E" x="500" y="0"></use><g transform="translate(778,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(1608,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">回</text></g><g transform="translate(2439,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">合</text></g><g transform="translate(3269,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">更</text></g><g transform="translate(4100,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">新</text></g><g transform="translate(4931,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(5761,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">对</text></g><g transform="translate(6592,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">于</text></g><g transform="translate(7423,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">每</text></g><g transform="translate(8253,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">个</text></g><g transform="translate(9084,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">回</text></g><g transform="translate(9915,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">合</text></g><g transform="translate(10745,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">执</text></g><g transform="translate(11576,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">行</text></g><g transform="translate(12407,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">以</text></g><g transform="translate(13237,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">下</text></g><g transform="translate(14068,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">操</text></g><g transform="translate(14899,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">作</text></g><g transform="translate(15729,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">：</text></g></g><g transform="translate(0,1478)"><g transform="translate(2000,0)"><use xlink:href="#E266-MJMAIN-32"></use><use xlink:href="#E266-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E266-MJMAIN-31" x="778" y="0"></use><g transform="translate(1278,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(2108,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">采</text></g><g transform="translate(2939,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">样</text></g><g transform="translate(3769,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(4600,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">用</text></g><g transform="translate(5431,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">策</text></g><g transform="translate(6261,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">略</text></g><use xlink:href="#E266-MJMATHI-3C0" x="7342" y="0"></use><g transform="translate(7915,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">生</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">成</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">轨</text></g><g transform="translate(2741,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">迹</text></g></g><g transform="translate(11738,0)"><use xlink:href="#E266-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E266-MJMAIN-30" x="866" y="-213"></use><use xlink:href="#E266-MJMAIN-2C" x="1066" y="0"></use><g transform="translate(1511,0)"><use xlink:href="#E266-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E266-MJMAIN-30" x="1060" y="-213"></use></g><use xlink:href="#E266-MJMAIN-2C" x="2714" y="0"></use><g transform="translate(3159,0)"><use xlink:href="#E266-MJMATHI-52" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E266-MJMAIN-31" x="1073" y="-213"></use></g><use xlink:href="#E266-MJMAIN-2C" x="4371" y="0"></use><g transform="translate(4816,0)"><use xlink:href="#E266-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E266-MJMAIN-31" x="866" y="-213"></use></g><use xlink:href="#E266-MJMAIN-2C" x="5883" y="0"></use><use xlink:href="#E266-MJMAIN-22EF" x="6327" y="0"></use><use xlink:href="#E266-MJMAIN-2C" x="7666" y="0"></use><g transform="translate(8111,0)"><use xlink:href="#E266-MJMATHI-53" x="0" y="0"></use><g transform="translate(613,-150)"><use transform="scale(0.707)" xlink:href="#E266-MJMATHI-54" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E266-MJMAIN-2212" x="704" y="0"></use><use transform="scale(0.707)" xlink:href="#E266-MJMAIN-31" x="1482" y="0"></use></g></g><use xlink:href="#E266-MJMAIN-2C" x="10225" y="0"></use><g transform="translate(10670,0)"><use xlink:href="#E266-MJMATHI-41" x="0" y="0"></use><g transform="translate(750,-150)"><use transform="scale(0.707)" xlink:href="#E266-MJMATHI-54" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E266-MJMAIN-2212" x="704" y="0"></use><use transform="scale(0.707)" xlink:href="#E266-MJMAIN-31" x="1482" y="0"></use></g></g><use xlink:href="#E266-MJMAIN-2C" x="12921" y="0"></use><g transform="translate(13366,0)"><use xlink:href="#E266-MJMATHI-52" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E266-MJMATHI-54" x="1073" y="-213"></use></g><use xlink:href="#E266-MJMAIN-2C" x="14723" y="0"></use><g transform="translate(15167,0)"><use xlink:href="#E266-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E266-MJMATHI-54" x="866" y="-213"></use></g></g><g transform="translate(28116,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">。</text></g></g></g></g><g transform="translate(0,170)"><g transform="translate(2000,0)"><use xlink:href="#E266-MJMAIN-32"></use><use xlink:href="#E266-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E266-MJMAIN-32" x="778" y="0"></use><g transform="translate(1278,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(2108,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">初</text></g><g transform="translate(2939,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">始</text></g><g transform="translate(3769,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">化</text></g><g transform="translate(4600,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">回</text></g><g transform="translate(5431,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">报</text></g><g transform="translate(6261,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(7092,0)"><use xlink:href="#E266-MJMATHI-47" x="0" y="0"></use><use xlink:href="#E266-MJMAIN-2190" x="1063" y="0"></use><use xlink:href="#E266-MJMAIN-30" x="2341" y="0"></use></g><g transform="translate(9934,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">。</text></g></g></g></g><g transform="translate(0,-1130)"><g transform="translate(2000,0)"><use xlink:href="#E266-MJMAIN-32"></use><use xlink:href="#E266-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E266-MJMAIN-33" x="778" y="0"></use><g transform="translate(1278,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(2108,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">逐</text></g><g transform="translate(2939,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">步</text></g><g transform="translate(3769,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">更</text></g><g transform="translate(4600,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">新</text></g><g transform="translate(5431,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(6261,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">对</text></g><g transform="translate(7342,0)"><use xlink:href="#E266-MJMATHI-74" x="0" y="0"></use><use xlink:href="#E266-MJMAIN-2190" x="638" y="0"></use><use xlink:href="#E266-MJMATHI-54" x="1916" y="0"></use><use xlink:href="#E266-MJMAIN-2212" x="2842" y="0"></use><use xlink:href="#E266-MJMAIN-31" x="3843" y="0"></use><use xlink:href="#E266-MJMAIN-2C" x="4343" y="0"></use><use xlink:href="#E266-MJMATHI-54" x="4787" y="0"></use><use xlink:href="#E266-MJMAIN-2212" x="5713" y="0"></use><use xlink:href="#E266-MJMAIN-32" x="6714" y="0"></use><use xlink:href="#E266-MJMAIN-2C" x="7214" y="0"></use><use xlink:href="#E266-MJMAIN-22EF" x="7658" y="0"></use><use xlink:href="#E266-MJMAIN-2C" x="8997" y="0"></use><use xlink:href="#E266-MJMAIN-30" x="9442" y="0"></use></g><g transform="translate(17284,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">，</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">执</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">行</text></g><g transform="translate(2741,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">以</text></g><g transform="translate(3572,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">下</text></g><g transform="translate(4403,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">步</text></g><g transform="translate(5233,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">骤</text></g><g transform="translate(6064,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">：</text></g></g></g></g><g transform="translate(0,-2430)"><g transform="translate(4000,0)"><use xlink:href="#E266-MJMAIN-32"></use><use xlink:href="#E266-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E266-MJMAIN-33" x="778" y="0"></use><use xlink:href="#E266-MJMAIN-2E" x="1278" y="0"></use><use xlink:href="#E266-MJMAIN-31" x="1556" y="0"></use><g transform="translate(2056,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(2886,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">更</text></g><g transform="translate(3717,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">新</text></g><g transform="translate(4547,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">回</text></g><g transform="translate(5378,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">报</text></g><g transform="translate(6209,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(7039,0)"><use xlink:href="#E266-MJMATHI-47" x="0" y="0"></use><use xlink:href="#E266-MJMAIN-2190" x="1063" y="0"></use><use xlink:href="#E266-MJMATHI-3B3" x="2341" y="0"></use><use xlink:href="#E266-MJMATHI-47" x="2884" y="0"></use><use xlink:href="#E266-MJMAIN-2B" x="3892" y="0"></use><g transform="translate(4893,0)"><use xlink:href="#E266-MJMATHI-52" x="0" y="0"></use><g transform="translate(759,-150)"><use transform="scale(0.707)" xlink:href="#E266-MJMATHI-74" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E266-MJMAIN-2B" x="361" y="0"></use><use transform="scale(0.707)" xlink:href="#E266-MJMAIN-31" x="1139" y="0"></use></g></g></g><g transform="translate(13950,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">；</text></g></g></g></g><g transform="translate(0,-3829)"><g transform="translate(4000,0)"><use xlink:href="#E266-MJMAIN-32"></use><use xlink:href="#E266-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E266-MJMAIN-33" x="778" y="0"></use><use xlink:href="#E266-MJMAIN-2E" x="1278" y="0"></use><use xlink:href="#E266-MJMAIN-32" x="1556" y="0"></use><g transform="translate(2056,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(2886,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">更</text></g><g transform="translate(3717,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">新</text></g><g transform="translate(4547,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">价</text></g><g transform="translate(5378,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">值</text></g><g transform="translate(6209,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(7039,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">更</text></g><g transform="translate(7870,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">新</text></g><use xlink:href="#E266-MJMAINB-77" x="8951" y="0"></use><g transform="translate(9782,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">以</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">减</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">小</text></g></g><g transform="translate(12774,0)"><use xlink:href="#E266-MJMAIN-5B" x="0" y="0"></use><use xlink:href="#E266-MJMATHI-47" x="278" y="0"></use><use xlink:href="#E266-MJMAIN-2212" x="1286" y="0"></use><use xlink:href="#E266-MJMATHI-71" x="2286" y="0"></use><use xlink:href="#E266-MJMAIN-28" x="2746" y="0"></use><g transform="translate(3135,0)"><use xlink:href="#E266-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E266-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E266-MJMAIN-2C" x="4103" y="0"></use><g transform="translate(4548,0)"><use xlink:href="#E266-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E266-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E266-MJMAIN-3B" x="5653" y="0"></use><use xlink:href="#E266-MJMAINB-77" x="6098" y="0"></use><use xlink:href="#E266-MJMAIN-29" x="6929" y="0"></use><g transform="translate(7318,0)"><use xlink:href="#E266-MJMAIN-5D" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E266-MJMAIN-32" x="393" y="583"></use></g></g><g transform="translate(20824,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">或</text></g></g><g transform="translate(22154,0)"><use xlink:href="#E266-MJMAIN-5B" x="0" y="0"></use><use xlink:href="#E266-MJMATHI-47" x="278" y="0"></use><use xlink:href="#E266-MJMAIN-2212" x="1286" y="0"></use><use xlink:href="#E266-MJMATHI-76" x="2286" y="0"></use><use xlink:href="#E266-MJMAIN-28" x="2771" y="0"></use><g transform="translate(3160,0)"><use xlink:href="#E266-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E266-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E266-MJMAIN-3B" x="4128" y="0"></use><use xlink:href="#E266-MJMAINB-77" x="4573" y="0"></use><use xlink:href="#E266-MJMAIN-29" x="5404" y="0"></use><g transform="translate(5793,0)"><use xlink:href="#E266-MJMAIN-5D" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E266-MJMAIN-32" x="393" y="583"></use></g></g><g transform="translate(28679,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">，</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">如</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">式</text></g></g><g transform="translate(31671,0)"><a class="mjx-svg-href" xlink:href="#mjx-eqn-eq%3A1_1" style="cursor: pointer;"><rect width="1278" height="1000" y="-250" fill="none" stroke="none" pointer-events="all"></rect><g class="MathJax_ref"><use xlink:href="#E266-MJMAIN-28"></use><use xlink:href="#E266-MJMAIN-31" x="389" y="0"></use><use xlink:href="#E266-MJMAIN-29" x="889" y="0"></use></g></a></g><g transform="translate(32949,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">或</text></g></g><g transform="translate(34280,0)"><a class="mjx-svg-href" xlink:href="#mjx-eqn-eq%3A2" style="cursor: pointer;"><rect width="1278" height="1000" y="-250" fill="none" stroke="none" pointer-events="all"></rect><g class="MathJax_ref"><use xlink:href="#E266-MJMAIN-28"></use><use xlink:href="#E266-MJMAIN-32" x="389" y="0"></use><use xlink:href="#E266-MJMAIN-29" x="889" y="0"></use></g></a></g><g transform="translate(35558,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">。</text></g></g></g></g><g transform="translate(0,-5179)"><g><rect fill="black" stroke="none" width="41597" height="100" x="0" y="-500"></rect></g></g></g></g></g></svg></span></div><script type="math/tex; mode=display" id="MathJax-Element-239">\; \\ \; \\
\large \textbf{算法 6-1   随机梯度下降函数近似评估策略的价值} \\
\begin{split}
\rule[5pt]{10mm}{0.1em} &\rule[5pt]{265mm}{0.1em} \\
&\text{1.（初始化）任意初始化参数 $\bold w$ 。} \\
&\text{2.（回合更新）对于每个回合执行以下操作：} \\
&\qquad \text{2.1（采样）用策略 $\pi$ 生成轨迹 $S_0,A_0,R_1,S_1,\cdots,S_{T-1},A_{T-1},R_T,S_T$ 。} \\
&\qquad \text{2.2（初始化回报）$G \leftarrow 0$ 。} \\
&\qquad \text{2.3（逐步更新）对 $t \leftarrow T-1,T-2,\cdots,0$ ，执行以下步骤：} \\
&\qquad \qquad \text{2.3.1（更新回报）$G \leftarrow \gamma G + R_{t+1}$ ；} \\
&\qquad \qquad \text{2.3.2（更新价值）更新 $\bold w$ 以减小 $[G-q(S_t,A_t;\bold w)]^2$ 或 $[G-v(S_t;\bold w)]^2$ ，如式 $\eqref{eq:1}$ 或 $\eqref{eq:2}$ 。} \\
\rule[-5pt]{10mm}{0.1em} &\rule[-5pt]{265mm}{0.1em}
\end{split}
\; \\ \; \\</script></div></div><p><span>将策略改进引入算法 6-1 即可实现最优策略求解算法 6-2 ：</span></p><div contenteditable="false" spellcheck="false" class="mathjax-block md-end-block md-math-block md-rawblock" id="mathjax-n13" cid="n13" mdtype="math_block"><div class="md-rawblock-container md-math-container" tabindex="-1"><div class="MathJax_SVG_Display"><span class="MathJax_SVG" id="MathJax-Element-240-Frame" tabindex="-1" style="font-size: 100%; display: inline-block; zoom: 0.971345;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="101.294ex" height="29.72ex" viewBox="-18.1 -43.5 43612.5 12796" role="img" focusable="false" style="vertical-align: -29.619ex; margin-left: -0.042ex; max-width: 100%;"><defs><path stroke-width="0" id="E267-MJMAINB-36" d="M48 318Q48 395 68 456T120 553T193 613T273 646T350 655Q425 655 461 616T497 524Q497 485 475 468T428 451Q399 451 378 470T357 521Q357 565 403 588Q375 601 351 601Q313 601 282 584Q242 565 222 526Q199 473 199 367Q201 369 210 380T227 396T246 410T275 422T312 426Q438 426 494 332Q526 285 526 208V199Q526 112 465 53Q428 17 388 3T285 -11Q236 -11 195 7T135 43T104 80Q48 165 48 318ZM375 231V244V268Q375 295 373 310T364 342T341 366T299 374H297Q231 374 208 287Q200 257 200 196Q201 120 209 100Q231 47 288 47Q351 47 368 90Q375 112 375 231Z"></path><path stroke-width="0" id="E267-MJMAINB-2D" d="M13 166V278H318V166H13Z"></path><path stroke-width="0" id="E267-MJMAINB-32" d="M175 580Q175 578 185 572T205 551T215 510Q215 467 191 449T137 430Q107 430 83 448T58 511Q58 558 91 592T168 640T259 654Q328 654 383 637Q451 610 484 563T517 459Q517 401 482 360T368 262Q340 243 265 184L210 140H274Q416 140 429 145Q439 148 447 186T455 237H517V233Q516 230 501 119Q489 9 486 4V0H57V25Q57 51 58 54Q60 57 109 106T215 214T288 291Q364 377 364 458Q364 515 328 553T231 592Q214 592 201 589T181 584T175 580Z"></path><path stroke-width="0" id="E267-MJMAIN-22EF" d="M78 250Q78 274 95 292T138 310Q162 310 180 294T199 251Q199 226 182 208T139 190T96 207T78 250ZM525 250Q525 274 542 292T585 310Q609 310 627 294T646 251Q646 226 629 208T586 190T543 207T525 250ZM972 250Q972 274 989 292T1032 310Q1056 310 1074 294T1093 251Q1093 226 1076 208T1033 190T990 207T972 250Z"></path><path stroke-width="0" id="E267-MJMAIN-36" d="M42 313Q42 476 123 571T303 666Q372 666 402 630T432 550Q432 525 418 510T379 495Q356 495 341 509T326 548Q326 592 373 601Q351 623 311 626Q240 626 194 566Q147 500 147 364L148 360Q153 366 156 373Q197 433 263 433H267Q313 433 348 414Q372 400 396 374T435 317Q456 268 456 210V192Q456 169 451 149Q440 90 387 34T253 -22Q225 -22 199 -14T143 16T92 75T56 172T42 313ZM257 397Q227 397 205 380T171 335T154 278T148 216Q148 133 160 97T198 39Q222 21 251 21Q302 21 329 59Q342 77 347 104T352 209Q352 289 347 316T329 361Q302 397 257 397Z"></path><path stroke-width="0" id="E267-MJMAIN-2D" d="M11 179V252H277V179H11Z"></path><path stroke-width="0" id="E267-MJMAIN-31" d="M213 578L200 573Q186 568 160 563T102 556H83V602H102Q149 604 189 617T245 641T273 663Q275 666 285 666Q294 666 302 660V361L303 61Q310 54 315 52T339 48T401 46H427V0H416Q395 3 257 3Q121 3 100 0H88V46H114Q136 46 152 46T177 47T193 50T201 52T207 57T213 61V578Z"></path><path stroke-width="0" id="E267-MJMAIN-32" d="M109 429Q82 429 66 447T50 491Q50 562 103 614T235 666Q326 666 387 610T449 465Q449 422 429 383T381 315T301 241Q265 210 201 149L142 93L218 92Q375 92 385 97Q392 99 409 186V189H449V186Q448 183 436 95T421 3V0H50V19V31Q50 38 56 46T86 81Q115 113 136 137Q145 147 170 174T204 211T233 244T261 278T284 308T305 340T320 369T333 401T340 431T343 464Q343 527 309 573T212 619Q179 619 154 602T119 569T109 550Q109 549 114 549Q132 549 151 535T170 489Q170 464 154 447T109 429Z"></path><path stroke-width="0" id="E267-MJMAIN-2E" d="M78 60Q78 84 95 102T138 120Q162 120 180 104T199 61Q199 36 182 18T139 0T96 17T78 60Z"></path><path stroke-width="0" id="E267-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E267-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E267-MJMAIN-22C5" d="M78 250Q78 274 95 292T138 310Q162 310 180 294T199 251Q199 226 182 208T139 190T96 207T78 250Z"></path><path stroke-width="0" id="E267-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E267-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E267-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E267-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E267-MJMATHI-3B5" d="M190 -22Q124 -22 76 11T27 107Q27 174 97 232L107 239L99 248Q76 273 76 304Q76 364 144 408T290 452H302Q360 452 405 421Q428 405 428 392Q428 381 417 369T391 356Q382 356 371 365T338 383T283 392Q217 392 167 368T116 308Q116 289 133 272Q142 263 145 262T157 264Q188 278 238 278H243Q308 278 308 247Q308 206 223 206Q177 206 142 219L132 212Q68 169 68 112Q68 39 201 39Q253 39 286 49T328 72T345 94T362 105Q376 103 376 88Q376 79 365 62T334 26T275 -8T190 -22Z"></path><path stroke-width="0" id="E267-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E267-MJMAIN-30" d="M96 585Q152 666 249 666Q297 666 345 640T423 548Q460 465 460 320Q460 165 417 83Q397 41 362 16T301 -15T250 -22Q224 -22 198 -16T137 16T82 83Q39 165 39 320Q39 494 96 585ZM321 597Q291 629 250 629Q208 629 178 597Q153 571 145 525T137 333Q137 175 145 125T181 46Q209 16 250 16Q290 16 318 46Q347 76 354 130T362 333Q362 478 354 524T321 597Z"></path><path stroke-width="0" id="E267-MJMATHI-41" d="M208 74Q208 50 254 46Q272 46 272 35Q272 34 270 22Q267 8 264 4T251 0Q249 0 239 0T205 1T141 2Q70 2 50 0H42Q35 7 35 11Q37 38 48 46H62Q132 49 164 96Q170 102 345 401T523 704Q530 716 547 716H555H572Q578 707 578 706L606 383Q634 60 636 57Q641 46 701 46Q726 46 726 36Q726 34 723 22Q720 7 718 4T704 0Q701 0 690 0T651 1T578 2Q484 2 455 0H443Q437 6 437 9T439 27Q443 40 445 43L449 46H469Q523 49 533 63L521 213H283L249 155Q208 86 208 74ZM516 260Q516 271 504 416T490 562L463 519Q447 492 400 412L310 260L413 259Q516 259 516 260Z"></path><path stroke-width="0" id="E267-MJMATHI-52" d="M230 637Q203 637 198 638T193 649Q193 676 204 682Q206 683 378 683Q550 682 564 680Q620 672 658 652T712 606T733 563T739 529Q739 484 710 445T643 385T576 351T538 338L545 333Q612 295 612 223Q612 212 607 162T602 80V71Q602 53 603 43T614 25T640 16Q668 16 686 38T712 85Q717 99 720 102T735 105Q755 105 755 93Q755 75 731 36Q693 -21 641 -21H632Q571 -21 531 4T487 82Q487 109 502 166T517 239Q517 290 474 313Q459 320 449 321T378 323H309L277 193Q244 61 244 59Q244 55 245 54T252 50T269 48T302 46H333Q339 38 339 37T336 19Q332 6 326 0H311Q275 2 180 2Q146 2 117 2T71 2T50 1Q33 1 33 10Q33 12 36 24Q41 43 46 45Q50 46 61 46H67Q94 46 127 49Q141 52 146 61Q149 65 218 339T287 628Q287 635 230 637ZM630 554Q630 586 609 608T523 636Q521 636 500 636T462 637H440Q393 637 386 627Q385 624 352 494T319 361Q319 360 388 360Q466 361 492 367Q556 377 592 426Q608 449 619 486T630 554Z"></path><path stroke-width="0" id="E267-MJMATHI-54" d="M40 437Q21 437 21 445Q21 450 37 501T71 602L88 651Q93 669 101 677H569H659Q691 677 697 676T704 667Q704 661 687 553T668 444Q668 437 649 437Q640 437 637 437T631 442L629 445Q629 451 635 490T641 551Q641 586 628 604T573 629Q568 630 515 631Q469 631 457 630T439 622Q438 621 368 343T298 60Q298 48 386 46Q418 46 427 45T436 36Q436 31 433 22Q429 4 424 1L422 0Q419 0 415 0Q410 0 363 1T228 2Q99 2 64 0H49Q43 6 43 9T45 27Q49 40 55 46H83H94Q174 46 189 55Q190 56 191 56Q196 59 201 76T241 233Q258 301 269 344Q339 619 339 625Q339 630 310 630H279Q212 630 191 624Q146 614 121 583T67 467Q60 445 57 441T43 437H40Z"></path><path stroke-width="0" id="E267-MJMAIN-2212" d="M84 237T84 250T98 270H679Q694 262 694 250T679 230H98Q84 237 84 250Z"></path><path stroke-width="0" id="E267-MJMAIN-33" d="M127 463Q100 463 85 480T69 524Q69 579 117 622T233 665Q268 665 277 664Q351 652 390 611T430 522Q430 470 396 421T302 350L299 348Q299 347 308 345T337 336T375 315Q457 262 457 175Q457 96 395 37T238 -22Q158 -22 100 21T42 130Q42 158 60 175T105 193Q133 193 151 175T169 130Q169 119 166 110T159 94T148 82T136 74T126 70T118 67L114 66Q165 21 238 21Q293 21 321 74Q338 107 338 175V195Q338 290 274 322Q259 328 213 329L171 330L168 332Q166 335 166 348Q166 366 174 366Q202 366 232 371Q266 376 294 413T322 525V533Q322 590 287 612Q265 626 240 626Q208 626 181 615T143 592T132 580H135Q138 579 143 578T153 573T165 566T175 555T183 540T186 520Q186 498 172 481T127 463Z"></path><path stroke-width="0" id="E267-MJMAIN-5B" d="M118 -250V750H255V710H158V-210H255V-250H118Z"></path><path stroke-width="0" id="E267-MJMATHI-47" d="M50 252Q50 367 117 473T286 641T490 704Q580 704 633 653Q642 643 648 636T656 626L657 623Q660 623 684 649Q691 655 699 663T715 679T725 690L740 705H746Q760 705 760 698Q760 694 728 561Q692 422 692 421Q690 416 687 415T669 413H653Q647 419 647 422Q647 423 648 429T650 449T651 481Q651 552 619 605T510 659Q492 659 471 656T418 643T357 615T294 567T236 496T189 394T158 260Q156 242 156 221Q156 173 170 136T206 79T256 45T308 28T353 24Q407 24 452 47T514 106Q517 114 529 161T541 214Q541 222 528 224T468 227H431Q425 233 425 235T427 254Q431 267 437 273H454Q494 271 594 271Q634 271 659 271T695 272T707 272Q721 272 721 263Q721 261 719 249Q714 230 709 228Q706 227 694 227Q674 227 653 224Q646 221 643 215T629 164Q620 131 614 108Q589 6 586 3Q584 1 581 1Q571 1 553 21T530 52Q530 53 528 52T522 47Q448 -22 322 -22Q201 -22 126 55T50 252Z"></path><path stroke-width="0" id="E267-MJMATHI-74" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path><path stroke-width="0" id="E267-MJMAIN-5D" d="M22 710V750H159V-250H22V-210H119V710H22Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><g transform="translate(12897,-2462)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">算</text><g transform="translate(1052,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">法</text></g><use transform="scale(1.2)" xlink:href="#E267-MJMAINB-36" x="1963" y="0"></use><use transform="scale(1.2)" xlink:href="#E267-MJMAINB-2D" x="2538" y="0"></use><use transform="scale(1.2)" xlink:href="#E267-MJMAINB-32" x="2921" y="0"></use><g transform="translate(4945,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">随</text></g><g transform="translate(5998,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">机</text></g><g transform="translate(7050,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">梯</text></g><g transform="translate(8103,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">度</text></g><g transform="translate(9156,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">下</text></g><g transform="translate(10209,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">降</text></g><g transform="translate(11262,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">求</text></g><g transform="translate(12315,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">最</text></g><g transform="translate(13368,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">优</text></g><g transform="translate(14420,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">策</text></g><g transform="translate(15473,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">略</text></g></g><g transform="translate(0,-7301)"><g transform="translate(-19,0)"><g transform="translate(0,3441)"><g><rect fill="black" stroke="none" width="1569" height="100" x="0" y="500"></rect></g></g><g transform="translate(0,-3242)"><g><rect fill="black" stroke="none" width="1569" height="100" x="0" y="-500"></rect></g></g></g><g transform="translate(1551,0)"><g transform="translate(0,3441)"><g><rect fill="black" stroke="none" width="41597" height="100" x="0" y="500"></rect></g></g><g transform="translate(0,2141)"><use xlink:href="#E267-MJMAIN-22EF" x="166" y="0"></use><g transform="translate(2505,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">同</text><g transform="translate(830,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">算</text></g><g transform="translate(1661,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">法</text></g><use xlink:href="#E267-MJMAIN-36" x="2741" y="0"></use><use xlink:href="#E267-MJMAIN-2D" x="3241" y="0"></use><use xlink:href="#E267-MJMAIN-31" x="3574" y="0"></use></g><use xlink:href="#E267-MJMAIN-22EF" x="7746" y="0"></use></g><g transform="translate(0,841)"><use xlink:href="#E267-MJMAIN-32"></use><use xlink:href="#E267-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E267-MJMAIN-31" x="778" y="0"></use><g transform="translate(1278,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(2108,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">采</text></g><g transform="translate(2939,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">样</text></g><g transform="translate(3769,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(4600,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">用</text></g><g transform="translate(5681,0)"><use xlink:href="#E267-MJMATHI-71" x="0" y="0"></use><use xlink:href="#E267-MJMAIN-28" x="460" y="0"></use><use xlink:href="#E267-MJMAIN-22C5" x="849" y="0"></use><use xlink:href="#E267-MJMAIN-2C" x="1127" y="0"></use><use xlink:href="#E267-MJMAIN-22C5" x="1571" y="0"></use><use xlink:href="#E267-MJMAIN-3B" x="1849" y="0"></use><use xlink:href="#E267-MJMAINB-77" x="2294" y="0"></use><use xlink:href="#E267-MJMAIN-29" x="3125" y="0"></use></g><g transform="translate(9195,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">导</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">出</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">策</text></g><g transform="translate(2741,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">略</text></g><g transform="translate(3572,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(4403,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">如</text></g></g><use xlink:href="#E267-MJMATHI-3B5" x="14679" y="0"></use><g transform="translate(15145,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">柔</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">性</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">策</text></g><g transform="translate(2741,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">略</text></g><g transform="translate(3572,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(4403,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">生</text></g><g transform="translate(5233,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">成</text></g><g transform="translate(6064,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">轨</text></g><g transform="translate(6895,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">迹</text></g></g><g transform="translate(23121,0)"><use xlink:href="#E267-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E267-MJMAIN-30" x="866" y="-213"></use><use xlink:href="#E267-MJMAIN-2C" x="1066" y="0"></use><g transform="translate(1511,0)"><use xlink:href="#E267-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E267-MJMAIN-30" x="1060" y="-213"></use></g><use xlink:href="#E267-MJMAIN-2C" x="2714" y="0"></use><g transform="translate(3159,0)"><use xlink:href="#E267-MJMATHI-52" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E267-MJMAIN-31" x="1073" y="-213"></use></g><use xlink:href="#E267-MJMAIN-2C" x="4371" y="0"></use><g transform="translate(4816,0)"><use xlink:href="#E267-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E267-MJMAIN-31" x="866" y="-213"></use></g><use xlink:href="#E267-MJMAIN-2C" x="5883" y="0"></use><use xlink:href="#E267-MJMAIN-22EF" x="6327" y="0"></use><use xlink:href="#E267-MJMAIN-2C" x="7666" y="0"></use><g transform="translate(8111,0)"><use xlink:href="#E267-MJMATHI-53" x="0" y="0"></use><g transform="translate(613,-150)"><use transform="scale(0.707)" xlink:href="#E267-MJMATHI-54" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E267-MJMAIN-2212" x="704" y="0"></use><use transform="scale(0.707)" xlink:href="#E267-MJMAIN-31" x="1482" y="0"></use></g></g><use xlink:href="#E267-MJMAIN-2C" x="10225" y="0"></use><g transform="translate(10670,0)"><use xlink:href="#E267-MJMATHI-41" x="0" y="0"></use><g transform="translate(750,-150)"><use transform="scale(0.707)" xlink:href="#E267-MJMATHI-54" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E267-MJMAIN-2212" x="704" y="0"></use><use transform="scale(0.707)" xlink:href="#E267-MJMAIN-31" x="1482" y="0"></use></g></g><use xlink:href="#E267-MJMAIN-2C" x="12921" y="0"></use><g transform="translate(13366,0)"><use xlink:href="#E267-MJMATHI-52" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E267-MJMATHI-54" x="1073" y="-213"></use></g><use xlink:href="#E267-MJMAIN-2C" x="14723" y="0"></use><g transform="translate(15167,0)"><use xlink:href="#E267-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E267-MJMATHI-54" x="866" y="-213"></use></g></g><g transform="translate(39500,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">。</text></g></g></g><g transform="translate(0,-509)"><use xlink:href="#E267-MJMAIN-22EF" x="166" y="0"></use><g transform="translate(2505,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">同</text><g transform="translate(830,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">算</text></g><g transform="translate(1661,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">法</text></g><use xlink:href="#E267-MJMAIN-36" x="2741" y="0"></use><use xlink:href="#E267-MJMAIN-2D" x="3241" y="0"></use><use xlink:href="#E267-MJMAIN-31" x="3574" y="0"></use></g><use xlink:href="#E267-MJMAIN-22EF" x="7746" y="0"></use></g><g transform="translate(0,-1892)"><g transform="translate(2000,0)"><use xlink:href="#E267-MJMAIN-32"></use><use xlink:href="#E267-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E267-MJMAIN-33" x="778" y="0"></use><use xlink:href="#E267-MJMAIN-2E" x="1278" y="0"></use><use xlink:href="#E267-MJMAIN-32" x="1556" y="0"></use><g transform="translate(2056,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(2886,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">更</text></g><g transform="translate(3717,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">新</text></g><g transform="translate(4547,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">价</text></g><g transform="translate(5378,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">值</text></g><g transform="translate(6209,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(7039,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">更</text></g><g transform="translate(7870,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">新</text></g><use xlink:href="#E267-MJMAINB-77" x="8951" y="0"></use><g transform="translate(9782,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">以</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">减</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">小</text></g></g><g transform="translate(12774,0)"><use xlink:href="#E267-MJMAIN-5B" x="0" y="0"></use><use xlink:href="#E267-MJMATHI-47" x="278" y="0"></use><use xlink:href="#E267-MJMAIN-2212" x="1286" y="0"></use><use xlink:href="#E267-MJMATHI-71" x="2286" y="0"></use><use xlink:href="#E267-MJMAIN-28" x="2746" y="0"></use><g transform="translate(3135,0)"><use xlink:href="#E267-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E267-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E267-MJMAIN-2C" x="4103" y="0"></use><g transform="translate(4548,0)"><use xlink:href="#E267-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E267-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E267-MJMAIN-3B" x="5653" y="0"></use><use xlink:href="#E267-MJMAINB-77" x="6098" y="0"></use><use xlink:href="#E267-MJMAIN-29" x="6929" y="0"></use><g transform="translate(7318,0)"><use xlink:href="#E267-MJMAIN-5D" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E267-MJMAIN-32" x="393" y="583"></use></g></g><g transform="translate(20824,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">，</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">如</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">式</text></g></g><g transform="translate(23815,0)"><a class="mjx-svg-href" xlink:href="#mjx-eqn-eq%3A1_1" style="cursor: pointer;"><rect width="1278" height="1000" y="-250" fill="none" stroke="none" pointer-events="all"></rect><g class="MathJax_ref"><use xlink:href="#E267-MJMAIN-28"></use><use xlink:href="#E267-MJMAIN-31" x="389" y="0"></use><use xlink:href="#E267-MJMAIN-29" x="889" y="0"></use></g></a></g><g transform="translate(25093,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">。</text></g></g></g></g><g transform="translate(0,-3242)"><g><rect fill="black" stroke="none" width="41597" height="100" x="0" y="-500"></rect></g></g></g></g></g></svg></span></div><script type="math/tex; mode=display" id="MathJax-Element-240">\; \\ \; \\
\large \textbf{算法 6-2   随机梯度下降求最优策略} \\
\begin{split}
\rule[5pt]{10mm}{0.1em} &\rule[5pt]{265mm}{0.1em} \\
&\cdots \quad \text{同算法 6-1} \quad \cdots \\
&\text{2.1（采样）用 $q(\cdot,\cdot;\bold w)$ 导出策略（如 $\varepsilon$ 柔性策略）生成轨迹 $S_0,A_0,R_1,S_1,\cdots,S_{T-1},A_{T-1},R_T,S_T$ 。} \\
&\cdots \quad \text{同算法 6-1} \quad \cdots \\
&\qquad \text{2.3.2（更新价值）更新 $\bold w$ 以减小 $[G-q(S_t,A_t;\bold w)]^2$ ，如式 $\eqref{eq:1}$ 。} \\
\rule[-5pt]{10mm}{0.1em} &\rule[-5pt]{265mm}{0.1em}
\end{split}
\; \\ \; \\</script></div></div><p><span>对于</span><strong><span>半梯度下降</span></strong><span>（semi-gradient descent）算法，就是在随机梯度下降算法的基础上，改用单步时序差分的回报估计 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="2.411ex" height="2.228ex" viewBox="0 -749.6 1038.3 959.2" role="img" focusable="false" style="vertical-align: -0.487ex;"><defs><path stroke-width="0" id="E340-MJMATHI-55" d="M107 637Q73 637 71 641Q70 643 70 649Q70 673 81 682Q83 683 98 683Q139 681 234 681Q268 681 297 681T342 682T362 682Q378 682 378 672Q378 670 376 658Q371 641 366 638H364Q362 638 359 638T352 638T343 637T334 637Q295 636 284 634T266 623Q265 621 238 518T184 302T154 169Q152 155 152 140Q152 86 183 55T269 24Q336 24 403 69T501 205L552 406Q599 598 599 606Q599 633 535 637Q511 637 511 648Q511 650 513 660Q517 676 519 679T529 683Q532 683 561 682T645 680Q696 680 723 681T752 682Q767 682 767 672Q767 650 759 642Q756 637 737 637Q666 633 648 597Q646 592 598 404Q557 235 548 205Q515 105 433 42T263 -22Q171 -22 116 34T60 167V183Q60 201 115 421Q164 622 164 628Q164 635 107 637Z"></path><path stroke-width="0" id="E340-MJMATHI-74" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E340-MJMATHI-55" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E340-MJMATHI-74" x="965" y="-213"></use></g></svg></span><script type="math/tex">U_t</script><span> ，并在对回合损失函数 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="22.636ex" height="7.051ex" viewBox="0 -1787.9 9746.1 3035.9" role="img" focusable="false" style="vertical-align: -2.899ex;"><defs><path stroke-width="0" id="E304-MJSZ2-2211" d="M60 948Q63 950 665 950H1267L1325 815Q1384 677 1388 669H1348L1341 683Q1320 724 1285 761Q1235 809 1174 838T1033 881T882 898T699 902H574H543H251L259 891Q722 258 724 252Q725 250 724 246Q721 243 460 -56L196 -356Q196 -357 407 -357Q459 -357 548 -357T676 -358Q812 -358 896 -353T1063 -332T1204 -283T1307 -196Q1328 -170 1348 -124H1388Q1388 -125 1381 -145T1356 -210T1325 -294L1267 -449L666 -450Q64 -450 61 -448Q55 -446 55 -439Q55 -437 57 -433L590 177Q590 178 557 222T452 366T322 544L56 909L55 924Q55 945 60 948Z"></path><path stroke-width="0" id="E304-MJMATHI-74" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path><path stroke-width="0" id="E304-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E304-MJMAIN-30" d="M96 585Q152 666 249 666Q297 666 345 640T423 548Q460 465 460 320Q460 165 417 83Q397 41 362 16T301 -15T250 -22Q224 -22 198 -16T137 16T82 83Q39 165 39 320Q39 494 96 585ZM321 597Q291 629 250 629Q208 629 178 597Q153 571 145 525T137 333Q137 175 145 125T181 46Q209 16 250 16Q290 16 318 46Q347 76 354 130T362 333Q362 478 354 524T321 597Z"></path><path stroke-width="0" id="E304-MJMATHI-54" d="M40 437Q21 437 21 445Q21 450 37 501T71 602L88 651Q93 669 101 677H569H659Q691 677 697 676T704 667Q704 661 687 553T668 444Q668 437 649 437Q640 437 637 437T631 442L629 445Q629 451 635 490T641 551Q641 586 628 604T573 629Q568 630 515 631Q469 631 457 630T439 622Q438 621 368 343T298 60Q298 48 386 46Q418 46 427 45T436 36Q436 31 433 22Q429 4 424 1L422 0Q419 0 415 0Q410 0 363 1T228 2Q99 2 64 0H49Q43 6 43 9T45 27Q49 40 55 46H83H94Q174 46 189 55Q190 56 191 56Q196 59 201 76T241 233Q258 301 269 344Q339 619 339 625Q339 630 310 630H279Q212 630 191 624Q146 614 121 583T67 467Q60 445 57 441T43 437H40Z"></path><path stroke-width="0" id="E304-MJMAIN-2212" d="M84 237T84 250T98 270H679Q694 262 694 250T679 230H98Q84 237 84 250Z"></path><path stroke-width="0" id="E304-MJMAIN-31" d="M213 578L200 573Q186 568 160 563T102 556H83V602H102Q149 604 189 617T245 641T273 663Q275 666 285 666Q294 666 302 660V361L303 61Q310 54 315 52T339 48T401 46H427V0H416Q395 3 257 3Q121 3 100 0H88V46H114Q136 46 152 46T177 47T193 50T201 52T207 57T213 61V578Z"></path><path stroke-width="0" id="E304-MJMAIN-5B" d="M118 -250V750H255V710H158V-210H255V-250H118Z"></path><path stroke-width="0" id="E304-MJMATHI-55" d="M107 637Q73 637 71 641Q70 643 70 649Q70 673 81 682Q83 683 98 683Q139 681 234 681Q268 681 297 681T342 682T362 682Q378 682 378 672Q378 670 376 658Q371 641 366 638H364Q362 638 359 638T352 638T343 637T334 637Q295 636 284 634T266 623Q265 621 238 518T184 302T154 169Q152 155 152 140Q152 86 183 55T269 24Q336 24 403 69T501 205L552 406Q599 598 599 606Q599 633 535 637Q511 637 511 648Q511 650 513 660Q517 676 519 679T529 683Q532 683 561 682T645 680Q696 680 723 681T752 682Q767 682 767 672Q767 650 759 642Q756 637 737 637Q666 633 648 597Q646 592 598 404Q557 235 548 205Q515 105 433 42T263 -22Q171 -22 116 34T60 167V183Q60 201 115 421Q164 622 164 628Q164 635 107 637Z"></path><path stroke-width="0" id="E304-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E304-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E304-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E304-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E304-MJMATHI-41" d="M208 74Q208 50 254 46Q272 46 272 35Q272 34 270 22Q267 8 264 4T251 0Q249 0 239 0T205 1T141 2Q70 2 50 0H42Q35 7 35 11Q37 38 48 46H62Q132 49 164 96Q170 102 345 401T523 704Q530 716 547 716H555H572Q578 707 578 706L606 383Q634 60 636 57Q641 46 701 46Q726 46 726 36Q726 34 723 22Q720 7 718 4T704 0Q701 0 690 0T651 1T578 2Q484 2 455 0H443Q437 6 437 9T439 27Q443 40 445 43L449 46H469Q523 49 533 63L521 213H283L249 155Q208 86 208 74ZM516 260Q516 271 504 416T490 562L463 519Q447 492 400 412L310 260L413 259Q516 259 516 260Z"></path><path stroke-width="0" id="E304-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E304-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E304-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E304-MJMAIN-5D" d="M22 710V750H159V-250H22V-210H119V710H22Z"></path><path stroke-width="0" id="E304-MJMAIN-32" d="M109 429Q82 429 66 447T50 491Q50 562 103 614T235 666Q326 666 387 610T449 465Q449 422 429 383T381 315T301 241Q265 210 201 149L142 93L218 92Q375 92 385 97Q392 99 409 186V189H449V186Q448 183 436 95T421 3V0H50V19V31Q50 38 56 46T86 81Q115 113 136 137Q145 147 170 174T204 211T233 244T261 278T284 308T305 340T320 369T333 401T340 431T343 464Q343 527 309 573T212 619Q179 619 154 602T119 569T109 550Q109 549 114 549Q132 549 151 535T170 489Q170 464 154 447T109 429Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E304-MJSZ2-2211" x="0" y="0"></use><g transform="translate(142,-1088)"><use transform="scale(0.707)" xlink:href="#E304-MJMATHI-74" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E304-MJMAIN-3D" x="361" y="0"></use><use transform="scale(0.707)" xlink:href="#E304-MJMAIN-30" x="1139" y="0"></use></g><g transform="translate(21,1150)"><use transform="scale(0.707)" xlink:href="#E304-MJMATHI-54" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E304-MJMAIN-2212" x="704" y="0"></use><use transform="scale(0.707)" xlink:href="#E304-MJMAIN-31" x="1482" y="0"></use></g><use xlink:href="#E304-MJMAIN-5B" x="1444" y="0"></use><g transform="translate(1722,0)"><use xlink:href="#E304-MJMATHI-55" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E304-MJMATHI-74" x="965" y="-213"></use></g><use xlink:href="#E304-MJMAIN-2212" x="2982" y="0"></use><use xlink:href="#E304-MJMATHI-71" x="3982" y="0"></use><use xlink:href="#E304-MJMAIN-28" x="4442" y="0"></use><g transform="translate(4831,0)"><use xlink:href="#E304-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E304-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E304-MJMAIN-2C" x="5799" y="0"></use><g transform="translate(6244,0)"><use xlink:href="#E304-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E304-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E304-MJMAIN-3B" x="7349" y="0"></use><use xlink:href="#E304-MJMAINB-77" x="7794" y="0"></use><use xlink:href="#E304-MJMAIN-29" x="8625" y="0"></use><g transform="translate(9014,0)"><use xlink:href="#E304-MJMAIN-5D" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E304-MJMAIN-32" x="393" y="583"></use></g></g></svg></span><script type="math/tex">\displaystyle \sum_{t=0}^{T-1}[U_t-q(S_t,A_t;\bold w)]^2</script><span> 或 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="19.094ex" height="7.051ex" viewBox="0 -1787.9 8221.2 3035.9" role="img" focusable="false" style="vertical-align: -2.899ex;"><defs><path stroke-width="0" id="E305-MJSZ2-2211" d="M60 948Q63 950 665 950H1267L1325 815Q1384 677 1388 669H1348L1341 683Q1320 724 1285 761Q1235 809 1174 838T1033 881T882 898T699 902H574H543H251L259 891Q722 258 724 252Q725 250 724 246Q721 243 460 -56L196 -356Q196 -357 407 -357Q459 -357 548 -357T676 -358Q812 -358 896 -353T1063 -332T1204 -283T1307 -196Q1328 -170 1348 -124H1388Q1388 -125 1381 -145T1356 -210T1325 -294L1267 -449L666 -450Q64 -450 61 -448Q55 -446 55 -439Q55 -437 57 -433L590 177Q590 178 557 222T452 366T322 544L56 909L55 924Q55 945 60 948Z"></path><path stroke-width="0" id="E305-MJMATHI-74" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path><path stroke-width="0" id="E305-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E305-MJMAIN-30" d="M96 585Q152 666 249 666Q297 666 345 640T423 548Q460 465 460 320Q460 165 417 83Q397 41 362 16T301 -15T250 -22Q224 -22 198 -16T137 16T82 83Q39 165 39 320Q39 494 96 585ZM321 597Q291 629 250 629Q208 629 178 597Q153 571 145 525T137 333Q137 175 145 125T181 46Q209 16 250 16Q290 16 318 46Q347 76 354 130T362 333Q362 478 354 524T321 597Z"></path><path stroke-width="0" id="E305-MJMATHI-54" d="M40 437Q21 437 21 445Q21 450 37 501T71 602L88 651Q93 669 101 677H569H659Q691 677 697 676T704 667Q704 661 687 553T668 444Q668 437 649 437Q640 437 637 437T631 442L629 445Q629 451 635 490T641 551Q641 586 628 604T573 629Q568 630 515 631Q469 631 457 630T439 622Q438 621 368 343T298 60Q298 48 386 46Q418 46 427 45T436 36Q436 31 433 22Q429 4 424 1L422 0Q419 0 415 0Q410 0 363 1T228 2Q99 2 64 0H49Q43 6 43 9T45 27Q49 40 55 46H83H94Q174 46 189 55Q190 56 191 56Q196 59 201 76T241 233Q258 301 269 344Q339 619 339 625Q339 630 310 630H279Q212 630 191 624Q146 614 121 583T67 467Q60 445 57 441T43 437H40Z"></path><path stroke-width="0" id="E305-MJMAIN-5B" d="M118 -250V750H255V710H158V-210H255V-250H118Z"></path><path stroke-width="0" id="E305-MJMATHI-55" d="M107 637Q73 637 71 641Q70 643 70 649Q70 673 81 682Q83 683 98 683Q139 681 234 681Q268 681 297 681T342 682T362 682Q378 682 378 672Q378 670 376 658Q371 641 366 638H364Q362 638 359 638T352 638T343 637T334 637Q295 636 284 634T266 623Q265 621 238 518T184 302T154 169Q152 155 152 140Q152 86 183 55T269 24Q336 24 403 69T501 205L552 406Q599 598 599 606Q599 633 535 637Q511 637 511 648Q511 650 513 660Q517 676 519 679T529 683Q532 683 561 682T645 680Q696 680 723 681T752 682Q767 682 767 672Q767 650 759 642Q756 637 737 637Q666 633 648 597Q646 592 598 404Q557 235 548 205Q515 105 433 42T263 -22Q171 -22 116 34T60 167V183Q60 201 115 421Q164 622 164 628Q164 635 107 637Z"></path><path stroke-width="0" id="E305-MJMAIN-2212" d="M84 237T84 250T98 270H679Q694 262 694 250T679 230H98Q84 237 84 250Z"></path><path stroke-width="0" id="E305-MJMATHI-76" d="M173 380Q173 405 154 405Q130 405 104 376T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Q21 294 29 316T53 368T97 419T160 441Q202 441 225 417T249 361Q249 344 246 335Q246 329 231 291T200 202T182 113Q182 86 187 69Q200 26 250 26Q287 26 319 60T369 139T398 222T409 277Q409 300 401 317T383 343T365 361T357 383Q357 405 376 424T417 443Q436 443 451 425T467 367Q467 340 455 284T418 159T347 40T241 -11Q177 -11 139 22Q102 54 102 117Q102 148 110 181T151 298Q173 362 173 380Z"></path><path stroke-width="0" id="E305-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E305-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E305-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E305-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E305-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E305-MJMAIN-5D" d="M22 710V750H159V-250H22V-210H119V710H22Z"></path><path stroke-width="0" id="E305-MJMAIN-32" d="M109 429Q82 429 66 447T50 491Q50 562 103 614T235 666Q326 666 387 610T449 465Q449 422 429 383T381 315T301 241Q265 210 201 149L142 93L218 92Q375 92 385 97Q392 99 409 186V189H449V186Q448 183 436 95T421 3V0H50V19V31Q50 38 56 46T86 81Q115 113 136 137Q145 147 170 174T204 211T233 244T261 278T284 308T305 340T320 369T333 401T340 431T343 464Q343 527 309 573T212 619Q179 619 154 602T119 569T109 550Q109 549 114 549Q132 549 151 535T170 489Q170 464 154 447T109 429Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E305-MJSZ2-2211" x="0" y="0"></use><g transform="translate(142,-1088)"><use transform="scale(0.707)" xlink:href="#E305-MJMATHI-74" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E305-MJMAIN-3D" x="361" y="0"></use><use transform="scale(0.707)" xlink:href="#E305-MJMAIN-30" x="1139" y="0"></use></g><use transform="scale(0.707)" xlink:href="#E305-MJMATHI-54" x="669" y="1626"></use><use xlink:href="#E305-MJMAIN-5B" x="1444" y="0"></use><g transform="translate(1722,0)"><use xlink:href="#E305-MJMATHI-55" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E305-MJMATHI-74" x="965" y="-213"></use></g><use xlink:href="#E305-MJMAIN-2212" x="2982" y="0"></use><use xlink:href="#E305-MJMATHI-76" x="3982" y="0"></use><use xlink:href="#E305-MJMAIN-28" x="4467" y="0"></use><g transform="translate(4856,0)"><use xlink:href="#E305-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E305-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E305-MJMAIN-3B" x="5824" y="0"></use><use xlink:href="#E305-MJMAINB-77" x="6269" y="0"></use><use xlink:href="#E305-MJMAIN-29" x="7100" y="0"></use><g transform="translate(7489,0)"><use xlink:href="#E305-MJMAIN-5D" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E305-MJMAIN-32" x="393" y="583"></use></g></g></svg></span><script type="math/tex">\displaystyle \sum_{t=0}^{T}[U_t-v(S_t;\bold w)]^2</script><span> 求梯度时，不对回报 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="30.181ex" height="2.71ex" viewBox="0 -832.7 12994.4 1166.9" role="img" focusable="false" style="vertical-align: -0.776ex;"><defs><path stroke-width="0" id="E306-MJMATHI-55" d="M107 637Q73 637 71 641Q70 643 70 649Q70 673 81 682Q83 683 98 683Q139 681 234 681Q268 681 297 681T342 682T362 682Q378 682 378 672Q378 670 376 658Q371 641 366 638H364Q362 638 359 638T352 638T343 637T334 637Q295 636 284 634T266 623Q265 621 238 518T184 302T154 169Q152 155 152 140Q152 86 183 55T269 24Q336 24 403 69T501 205L552 406Q599 598 599 606Q599 633 535 637Q511 637 511 648Q511 650 513 660Q517 676 519 679T529 683Q532 683 561 682T645 680Q696 680 723 681T752 682Q767 682 767 672Q767 650 759 642Q756 637 737 637Q666 633 648 597Q646 592 598 404Q557 235 548 205Q515 105 433 42T263 -22Q171 -22 116 34T60 167V183Q60 201 115 421Q164 622 164 628Q164 635 107 637Z"></path><path stroke-width="0" id="E306-MJMATHI-74" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path><path stroke-width="0" id="E306-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E306-MJMATHI-52" d="M230 637Q203 637 198 638T193 649Q193 676 204 682Q206 683 378 683Q550 682 564 680Q620 672 658 652T712 606T733 563T739 529Q739 484 710 445T643 385T576 351T538 338L545 333Q612 295 612 223Q612 212 607 162T602 80V71Q602 53 603 43T614 25T640 16Q668 16 686 38T712 85Q717 99 720 102T735 105Q755 105 755 93Q755 75 731 36Q693 -21 641 -21H632Q571 -21 531 4T487 82Q487 109 502 166T517 239Q517 290 474 313Q459 320 449 321T378 323H309L277 193Q244 61 244 59Q244 55 245 54T252 50T269 48T302 46H333Q339 38 339 37T336 19Q332 6 326 0H311Q275 2 180 2Q146 2 117 2T71 2T50 1Q33 1 33 10Q33 12 36 24Q41 43 46 45Q50 46 61 46H67Q94 46 127 49Q141 52 146 61Q149 65 218 339T287 628Q287 635 230 637ZM630 554Q630 586 609 608T523 636Q521 636 500 636T462 637H440Q393 637 386 627Q385 624 352 494T319 361Q319 360 388 360Q466 361 492 367Q556 377 592 426Q608 449 619 486T630 554Z"></path><path stroke-width="0" id="E306-MJMAIN-2B" d="M56 237T56 250T70 270H369V420L370 570Q380 583 389 583Q402 583 409 568V270H707Q722 262 722 250T707 230H409V-68Q401 -82 391 -82H389H387Q375 -82 369 -68V230H70Q56 237 56 250Z"></path><path stroke-width="0" id="E306-MJMAIN-31" d="M213 578L200 573Q186 568 160 563T102 556H83V602H102Q149 604 189 617T245 641T273 663Q275 666 285 666Q294 666 302 660V361L303 61Q310 54 315 52T339 48T401 46H427V0H416Q395 3 257 3Q121 3 100 0H88V46H114Q136 46 152 46T177 47T193 50T201 52T207 57T213 61V578Z"></path><path stroke-width="0" id="E306-MJMATHI-3B3" d="M31 249Q11 249 11 258Q11 275 26 304T66 365T129 418T206 441Q233 441 239 440Q287 429 318 386T371 255Q385 195 385 170Q385 166 386 166L398 193Q418 244 443 300T486 391T508 430Q510 431 524 431H537Q543 425 543 422Q543 418 522 378T463 251T391 71Q385 55 378 6T357 -100Q341 -165 330 -190T303 -216Q286 -216 286 -188Q286 -138 340 32L346 51L347 69Q348 79 348 100Q348 257 291 317Q251 355 196 355Q148 355 108 329T51 260Q49 251 47 251Q45 249 31 249Z"></path><path stroke-width="0" id="E306-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E306-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E306-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E306-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E306-MJMATHI-41" d="M208 74Q208 50 254 46Q272 46 272 35Q272 34 270 22Q267 8 264 4T251 0Q249 0 239 0T205 1T141 2Q70 2 50 0H42Q35 7 35 11Q37 38 48 46H62Q132 49 164 96Q170 102 345 401T523 704Q530 716 547 716H555H572Q578 707 578 706L606 383Q634 60 636 57Q641 46 701 46Q726 46 726 36Q726 34 723 22Q720 7 718 4T704 0Q701 0 690 0T651 1T578 2Q484 2 455 0H443Q437 6 437 9T439 27Q443 40 445 43L449 46H469Q523 49 533 63L521 213H283L249 155Q208 86 208 74ZM516 260Q516 271 504 416T490 562L463 519Q447 492 400 412L310 260L413 259Q516 259 516 260Z"></path><path stroke-width="0" id="E306-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E306-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E306-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E306-MJMATHI-55" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E306-MJMATHI-74" x="965" y="-213"></use><use xlink:href="#E306-MJMAIN-3D" x="1316" y="0"></use><g transform="translate(2371,0)"><use xlink:href="#E306-MJMATHI-52" x="0" y="0"></use><g transform="translate(759,-150)"><use transform="scale(0.707)" xlink:href="#E306-MJMATHI-74" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E306-MJMAIN-2B" x="361" y="0"></use><use transform="scale(0.707)" xlink:href="#E306-MJMAIN-31" x="1139" y="0"></use></g></g><use xlink:href="#E306-MJMAIN-2B" x="4611" y="0"></use><use xlink:href="#E306-MJMATHI-3B3" x="5612" y="0"></use><use xlink:href="#E306-MJMATHI-71" x="6155" y="0"></use><use xlink:href="#E306-MJMAIN-28" x="6615" y="0"></use><g transform="translate(7004,0)"><use xlink:href="#E306-MJMATHI-53" x="0" y="0"></use><g transform="translate(613,-150)"><use transform="scale(0.707)" xlink:href="#E306-MJMATHI-74" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E306-MJMAIN-2B" x="361" y="0"></use><use transform="scale(0.707)" xlink:href="#E306-MJMAIN-31" x="1139" y="0"></use></g></g><use xlink:href="#E306-MJMAIN-2C" x="8876" y="0"></use><g transform="translate(9320,0)"><use xlink:href="#E306-MJMATHI-41" x="0" y="0"></use><g transform="translate(750,-150)"><use transform="scale(0.707)" xlink:href="#E306-MJMATHI-74" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E306-MJMAIN-2B" x="361" y="0"></use><use transform="scale(0.707)" xlink:href="#E306-MJMAIN-31" x="1139" y="0"></use></g></g><use xlink:href="#E306-MJMAIN-3B" x="11329" y="0"></use><use xlink:href="#E306-MJMAINB-77" x="11774" y="0"></use><use xlink:href="#E306-MJMAIN-29" x="12605" y="0"></use></g></svg></span><script type="math/tex">U_t=R_{t+1} + \gamma q(S_{t+1},A_{t+1};\bold w)</script><span> 或 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="24.54ex" height="2.71ex" viewBox="0 -832.7 10565.8 1166.9" role="img" focusable="false" style="vertical-align: -0.776ex;"><defs><path stroke-width="0" id="E307-MJMATHI-55" d="M107 637Q73 637 71 641Q70 643 70 649Q70 673 81 682Q83 683 98 683Q139 681 234 681Q268 681 297 681T342 682T362 682Q378 682 378 672Q378 670 376 658Q371 641 366 638H364Q362 638 359 638T352 638T343 637T334 637Q295 636 284 634T266 623Q265 621 238 518T184 302T154 169Q152 155 152 140Q152 86 183 55T269 24Q336 24 403 69T501 205L552 406Q599 598 599 606Q599 633 535 637Q511 637 511 648Q511 650 513 660Q517 676 519 679T529 683Q532 683 561 682T645 680Q696 680 723 681T752 682Q767 682 767 672Q767 650 759 642Q756 637 737 637Q666 633 648 597Q646 592 598 404Q557 235 548 205Q515 105 433 42T263 -22Q171 -22 116 34T60 167V183Q60 201 115 421Q164 622 164 628Q164 635 107 637Z"></path><path stroke-width="0" id="E307-MJMATHI-74" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path><path stroke-width="0" id="E307-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E307-MJMATHI-52" d="M230 637Q203 637 198 638T193 649Q193 676 204 682Q206 683 378 683Q550 682 564 680Q620 672 658 652T712 606T733 563T739 529Q739 484 710 445T643 385T576 351T538 338L545 333Q612 295 612 223Q612 212 607 162T602 80V71Q602 53 603 43T614 25T640 16Q668 16 686 38T712 85Q717 99 720 102T735 105Q755 105 755 93Q755 75 731 36Q693 -21 641 -21H632Q571 -21 531 4T487 82Q487 109 502 166T517 239Q517 290 474 313Q459 320 449 321T378 323H309L277 193Q244 61 244 59Q244 55 245 54T252 50T269 48T302 46H333Q339 38 339 37T336 19Q332 6 326 0H311Q275 2 180 2Q146 2 117 2T71 2T50 1Q33 1 33 10Q33 12 36 24Q41 43 46 45Q50 46 61 46H67Q94 46 127 49Q141 52 146 61Q149 65 218 339T287 628Q287 635 230 637ZM630 554Q630 586 609 608T523 636Q521 636 500 636T462 637H440Q393 637 386 627Q385 624 352 494T319 361Q319 360 388 360Q466 361 492 367Q556 377 592 426Q608 449 619 486T630 554Z"></path><path stroke-width="0" id="E307-MJMAIN-2B" d="M56 237T56 250T70 270H369V420L370 570Q380 583 389 583Q402 583 409 568V270H707Q722 262 722 250T707 230H409V-68Q401 -82 391 -82H389H387Q375 -82 369 -68V230H70Q56 237 56 250Z"></path><path stroke-width="0" id="E307-MJMAIN-31" d="M213 578L200 573Q186 568 160 563T102 556H83V602H102Q149 604 189 617T245 641T273 663Q275 666 285 666Q294 666 302 660V361L303 61Q310 54 315 52T339 48T401 46H427V0H416Q395 3 257 3Q121 3 100 0H88V46H114Q136 46 152 46T177 47T193 50T201 52T207 57T213 61V578Z"></path><path stroke-width="0" id="E307-MJMATHI-3B3" d="M31 249Q11 249 11 258Q11 275 26 304T66 365T129 418T206 441Q233 441 239 440Q287 429 318 386T371 255Q385 195 385 170Q385 166 386 166L398 193Q418 244 443 300T486 391T508 430Q510 431 524 431H537Q543 425 543 422Q543 418 522 378T463 251T391 71Q385 55 378 6T357 -100Q341 -165 330 -190T303 -216Q286 -216 286 -188Q286 -138 340 32L346 51L347 69Q348 79 348 100Q348 257 291 317Q251 355 196 355Q148 355 108 329T51 260Q49 251 47 251Q45 249 31 249Z"></path><path stroke-width="0" id="E307-MJMATHI-76" d="M173 380Q173 405 154 405Q130 405 104 376T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Q21 294 29 316T53 368T97 419T160 441Q202 441 225 417T249 361Q249 344 246 335Q246 329 231 291T200 202T182 113Q182 86 187 69Q200 26 250 26Q287 26 319 60T369 139T398 222T409 277Q409 300 401 317T383 343T365 361T357 383Q357 405 376 424T417 443Q436 443 451 425T467 367Q467 340 455 284T418 159T347 40T241 -11Q177 -11 139 22Q102 54 102 117Q102 148 110 181T151 298Q173 362 173 380Z"></path><path stroke-width="0" id="E307-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E307-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E307-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E307-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E307-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E307-MJMATHI-55" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E307-MJMATHI-74" x="965" y="-213"></use><use xlink:href="#E307-MJMAIN-3D" x="1316" y="0"></use><g transform="translate(2371,0)"><use xlink:href="#E307-MJMATHI-52" x="0" y="0"></use><g transform="translate(759,-150)"><use transform="scale(0.707)" xlink:href="#E307-MJMATHI-74" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E307-MJMAIN-2B" x="361" y="0"></use><use transform="scale(0.707)" xlink:href="#E307-MJMAIN-31" x="1139" y="0"></use></g></g><use xlink:href="#E307-MJMAIN-2B" x="4611" y="0"></use><use xlink:href="#E307-MJMATHI-3B3" x="5612" y="0"></use><use xlink:href="#E307-MJMATHI-76" x="6155" y="0"></use><use xlink:href="#E307-MJMAIN-28" x="6640" y="0"></use><g transform="translate(7029,0)"><use xlink:href="#E307-MJMATHI-53" x="0" y="0"></use><g transform="translate(613,-150)"><use transform="scale(0.707)" xlink:href="#E307-MJMATHI-74" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E307-MJMAIN-2B" x="361" y="0"></use><use transform="scale(0.707)" xlink:href="#E307-MJMAIN-31" x="1139" y="0"></use></g></g><use xlink:href="#E307-MJMAIN-3B" x="8901" y="0"></use><use xlink:href="#E307-MJMAINB-77" x="9345" y="0"></use><use xlink:href="#E307-MJMAIN-29" x="10176" y="0"></use></g></svg></span><script type="math/tex">U_t=R_{t+1} + \gamma v(S_{t+1};\bold w)</script><span> 求梯度。将半梯度下降算法与第五章节的算法相结合可以得到以下两个算法：</span></p><div contenteditable="false" spellcheck="false" class="mathjax-block md-end-block md-math-block md-rawblock" id="mathjax-n15" cid="n15" mdtype="math_block"><div class="md-rawblock-container md-math-container" tabindex="-1"><div class="MathJax_SVG_Display"><span class="MathJax_SVG" id="MathJax-Element-241-Frame" tabindex="-1" style="font-size: 100%; display: inline-block; zoom: 0.971345;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="101.294ex" height="45.154ex" viewBox="-18.1 -43.5 43612.5 19441.2" role="img" focusable="false" style="vertical-align: -45.053ex; margin-left: -0.042ex; max-width: 100%;"><defs><path stroke-width="0" id="E268-MJMAINB-36" d="M48 318Q48 395 68 456T120 553T193 613T273 646T350 655Q425 655 461 616T497 524Q497 485 475 468T428 451Q399 451 378 470T357 521Q357 565 403 588Q375 601 351 601Q313 601 282 584Q242 565 222 526Q199 473 199 367Q201 369 210 380T227 396T246 410T275 422T312 426Q438 426 494 332Q526 285 526 208V199Q526 112 465 53Q428 17 388 3T285 -11Q236 -11 195 7T135 43T104 80Q48 165 48 318ZM375 231V244V268Q375 295 373 310T364 342T341 366T299 374H297Q231 374 208 287Q200 257 200 196Q201 120 209 100Q231 47 288 47Q351 47 368 90Q375 112 375 231Z"></path><path stroke-width="0" id="E268-MJMAINB-2D" d="M13 166V278H318V166H13Z"></path><path stroke-width="0" id="E268-MJMAINB-33" d="M80 503Q80 565 133 610T274 655Q366 655 421 623T491 538Q493 528 493 510Q493 446 453 407T361 348L376 344Q452 324 489 281T526 184Q526 152 514 121T474 58T392 8T265 -11Q175 -11 111 34T48 152Q50 187 72 209T132 232Q171 232 193 208T216 147Q216 136 214 126T207 108T197 94T187 84T178 77T170 72L168 71Q168 70 179 65T215 54T266 48H270Q331 48 350 105Q358 128 358 185Q358 239 348 268T309 313Q292 321 242 322Q205 322 198 324T191 341V348Q191 366 196 369T232 375Q239 375 247 376T260 377T268 378Q284 383 297 393T326 436T341 517Q341 536 339 547T331 573T308 593T266 600Q248 600 241 599Q214 593 183 576Q234 556 234 503Q234 462 210 444T157 426Q126 426 103 446T80 503Z"></path><path stroke-width="0" id="E268-MJMAINB-53" d="M64 493Q64 582 120 636T264 696H272Q280 697 285 697Q380 697 454 645L480 669Q484 672 488 676T495 683T500 688T504 691T508 693T511 695T514 696T517 697T522 697Q536 697 539 691T542 652V577Q542 557 542 532T543 500Q543 472 540 465T524 458H511H505Q489 458 485 461T479 478Q472 529 449 564T393 614T336 634T287 639Q228 639 203 610T177 544Q177 517 195 493T247 457Q253 454 343 436T475 391Q574 326 574 207V200Q574 163 559 120Q517 12 389 -9Q380 -10 346 -10Q308 -10 275 -5T221 7T184 22T160 35T151 40L126 17Q122 14 118 10T111 3T106 -2T102 -5T98 -7T95 -9T92 -10T89 -11T84 -11Q70 -11 67 -4T64 35V108Q64 128 64 153T63 185Q63 203 63 211T69 223T77 227T94 228H100Q118 228 122 225T126 205Q130 125 193 88T345 51Q408 51 434 82T460 157Q460 196 439 221T388 257Q384 259 305 276T221 295Q155 313 110 366T64 493Z"></path><path stroke-width="0" id="E268-MJMAINB-41" d="M296 0Q278 3 164 3Q58 3 49 0H40V62H92Q144 62 144 64Q388 682 397 689Q403 698 434 698Q463 698 471 689Q475 686 538 530T663 218L724 64Q724 62 776 62H828V0H817Q796 3 658 3Q509 3 485 0H472V62H517Q561 62 561 63L517 175H262L240 120Q218 65 217 64Q217 62 261 62H306V0H296ZM390 237L492 238L440 365Q390 491 388 491Q287 239 287 237H390Z"></path><path stroke-width="0" id="E268-MJMAINB-52" d="M394 0Q370 3 222 3Q75 3 51 0H39V62H147V624H39V686H234Q256 686 299 686T362 687Q479 687 554 669T681 593Q716 550 716 497Q716 390 568 338Q569 337 572 336T577 332Q605 317 623 300T650 258T662 218T668 172Q678 98 689 76Q707 40 748 40Q770 40 780 54T795 88T801 111Q805 117 827 117H831Q846 117 852 113T858 92Q857 78 852 63T834 30T797 1T739 -11Q630 -11 580 12T511 87Q506 104 506 168Q506 170 506 178T507 194Q507 289 438 313Q424 318 356 318H298V62H406V0H394ZM366 369Q459 370 490 381Q548 402 548 476V498V517Q548 578 513 600Q479 624 392 624H358H298V369H366Z"></path><path stroke-width="0" id="E268-MJMAIN-31" d="M213 578L200 573Q186 568 160 563T102 556H83V602H102Q149 604 189 617T245 641T273 663Q275 666 285 666Q294 666 302 660V361L303 61Q310 54 315 52T339 48T401 46H427V0H416Q395 3 257 3Q121 3 100 0H88V46H114Q136 46 152 46T177 47T193 50T201 52T207 57T213 61V578Z"></path><path stroke-width="0" id="E268-MJMAIN-2E" d="M78 60Q78 84 95 102T138 120Q162 120 180 104T199 61Q199 36 182 18T139 0T96 17T78 60Z"></path><path stroke-width="0" id="E268-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E268-MJMAIN-32" d="M109 429Q82 429 66 447T50 491Q50 562 103 614T235 666Q326 666 387 610T449 465Q449 422 429 383T381 315T301 241Q265 210 201 149L142 93L218 92Q375 92 385 97Q392 99 409 186V189H449V186Q448 183 436 95T421 3V0H50V19V31Q50 38 56 46T86 81Q115 113 136 137Q145 147 170 174T204 211T233 244T261 278T284 308T305 340T320 369T333 401T340 431T343 464Q343 527 309 573T212 619Q179 619 154 602T119 569T109 550Q109 549 114 549Q132 549 151 535T170 489Q170 464 154 447T109 429Z"></path><path stroke-width="0" id="E268-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E268-MJMATHI-3C0" d="M132 -11Q98 -11 98 22V33L111 61Q186 219 220 334L228 358H196Q158 358 142 355T103 336Q92 329 81 318T62 297T53 285Q51 284 38 284Q19 284 19 294Q19 300 38 329T93 391T164 429Q171 431 389 431Q549 431 553 430Q573 423 573 402Q573 371 541 360Q535 358 472 358H408L405 341Q393 269 393 222Q393 170 402 129T421 65T431 37Q431 20 417 5T381 -10Q370 -10 363 -7T347 17T331 77Q330 86 330 121Q330 170 339 226T357 318T367 358H269L268 354Q268 351 249 275T206 114T175 17Q164 -11 132 -11Z"></path><path stroke-width="0" id="E268-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E268-MJMAIN-22C5" d="M78 250Q78 274 95 292T138 310Q162 310 180 294T199 251Q199 226 182 208T139 190T96 207T78 250Z"></path><path stroke-width="0" id="E268-MJMAIN-2223" d="M139 -249H137Q125 -249 119 -235V251L120 737Q130 750 139 750Q152 750 159 735V-235Q151 -249 141 -249H139Z"></path><path stroke-width="0" id="E268-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E268-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E268-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E268-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E268-MJMATHI-41" d="M208 74Q208 50 254 46Q272 46 272 35Q272 34 270 22Q267 8 264 4T251 0Q249 0 239 0T205 1T141 2Q70 2 50 0H42Q35 7 35 11Q37 38 48 46H62Q132 49 164 96Q170 102 345 401T523 704Q530 716 547 716H555H572Q578 707 578 706L606 383Q634 60 636 57Q641 46 701 46Q726 46 726 36Q726 34 723 22Q720 7 718 4T704 0Q701 0 690 0T651 1T578 2Q484 2 455 0H443Q437 6 437 9T439 27Q443 40 445 43L449 46H469Q523 49 533 63L521 213H283L249 155Q208 86 208 74ZM516 260Q516 271 504 416T490 562L463 519Q447 492 400 412L310 260L413 259Q516 259 516 260Z"></path><path stroke-width="0" id="E268-MJMATHI-52" d="M230 637Q203 637 198 638T193 649Q193 676 204 682Q206 683 378 683Q550 682 564 680Q620 672 658 652T712 606T733 563T739 529Q739 484 710 445T643 385T576 351T538 338L545 333Q612 295 612 223Q612 212 607 162T602 80V71Q602 53 603 43T614 25T640 16Q668 16 686 38T712 85Q717 99 720 102T735 105Q755 105 755 93Q755 75 731 36Q693 -21 641 -21H632Q571 -21 531 4T487 82Q487 109 502 166T517 239Q517 290 474 313Q459 320 449 321T378 323H309L277 193Q244 61 244 59Q244 55 245 54T252 50T269 48T302 46H333Q339 38 339 37T336 19Q332 6 326 0H311Q275 2 180 2Q146 2 117 2T71 2T50 1Q33 1 33 10Q33 12 36 24Q41 43 46 45Q50 46 61 46H67Q94 46 127 49Q141 52 146 61Q149 65 218 339T287 628Q287 635 230 637ZM630 554Q630 586 609 608T523 636Q521 636 500 636T462 637H440Q393 637 386 627Q385 624 352 494T319 361Q319 360 388 360Q466 361 492 367Q556 377 592 426Q608 449 619 486T630 554Z"></path><path stroke-width="0" id="E268-MJMAIN-2032" d="M79 43Q73 43 52 49T30 61Q30 68 85 293T146 528Q161 560 198 560Q218 560 240 545T262 501Q262 496 260 486Q259 479 173 263T84 45T79 43Z"></path><path stroke-width="0" id="E268-MJMAIN-33" d="M127 463Q100 463 85 480T69 524Q69 579 117 622T233 665Q268 665 277 664Q351 652 390 611T430 522Q430 470 396 421T302 350L299 348Q299 347 308 345T337 336T375 315Q457 262 457 175Q457 96 395 37T238 -22Q158 -22 100 21T42 130Q42 158 60 175T105 193Q133 193 151 175T169 130Q169 119 166 110T159 94T148 82T136 74T126 70T118 67L114 66Q165 21 238 21Q293 21 321 74Q338 107 338 175V195Q338 290 274 322Q259 328 213 329L171 330L168 332Q166 335 166 348Q166 366 174 366Q202 366 232 371Q266 376 294 413T322 525V533Q322 590 287 612Q265 626 240 626Q208 626 181 615T143 592T132 580H135Q138 579 143 578T153 573T165 566T175 555T183 540T186 520Q186 498 172 481T127 463Z"></path><path stroke-width="0" id="E268-MJMATHI-55" d="M107 637Q73 637 71 641Q70 643 70 649Q70 673 81 682Q83 683 98 683Q139 681 234 681Q268 681 297 681T342 682T362 682Q378 682 378 672Q378 670 376 658Q371 641 366 638H364Q362 638 359 638T352 638T343 637T334 637Q295 636 284 634T266 623Q265 621 238 518T184 302T154 169Q152 155 152 140Q152 86 183 55T269 24Q336 24 403 69T501 205L552 406Q599 598 599 606Q599 633 535 637Q511 637 511 648Q511 650 513 660Q517 676 519 679T529 683Q532 683 561 682T645 680Q696 680 723 681T752 682Q767 682 767 672Q767 650 759 642Q756 637 737 637Q666 633 648 597Q646 592 598 404Q557 235 548 205Q515 105 433 42T263 -22Q171 -22 116 34T60 167V183Q60 201 115 421Q164 622 164 628Q164 635 107 637Z"></path><path stroke-width="0" id="E268-MJMAIN-2190" d="M944 261T944 250T929 230H165Q167 228 182 216T211 189T244 152T277 96T303 25Q308 7 308 0Q308 -11 288 -11Q281 -11 278 -11T272 -7T267 2T263 21Q245 94 195 151T73 236Q58 242 55 247Q55 254 59 257T73 264Q121 283 158 314T215 375T247 434T264 480L267 497Q269 503 270 505T275 509T288 511Q308 511 308 500Q308 493 303 475Q293 438 278 406T246 352T215 315T185 287T165 270H929Q944 261 944 250Z"></path><path stroke-width="0" id="E268-MJMAIN-2B" d="M56 237T56 250T70 270H369V420L370 570Q380 583 389 583Q402 583 409 568V270H707Q722 262 722 250T707 230H409V-68Q401 -82 391 -82H389H387Q375 -82 369 -68V230H70Q56 237 56 250Z"></path><path stroke-width="0" id="E268-MJMATHI-3B3" d="M31 249Q11 249 11 258Q11 275 26 304T66 365T129 418T206 441Q233 441 239 440Q287 429 318 386T371 255Q385 195 385 170Q385 166 386 166L398 193Q418 244 443 300T486 391T508 430Q510 431 524 431H537Q543 425 543 422Q543 418 522 378T463 251T391 71Q385 55 378 6T357 -100Q341 -165 330 -190T303 -216Q286 -216 286 -188Q286 -138 340 32L346 51L347 69Q348 79 348 100Q348 257 291 317Q251 355 196 355Q148 355 108 329T51 260Q49 251 47 251Q45 249 31 249Z"></path><path stroke-width="0" id="E268-MJMAIN-34" d="M462 0Q444 3 333 3Q217 3 199 0H190V46H221Q241 46 248 46T265 48T279 53T286 61Q287 63 287 115V165H28V211L179 442Q332 674 334 675Q336 677 355 677H373L379 671V211H471V165H379V114Q379 73 379 66T385 54Q393 47 442 46H471V0H462ZM293 211V545L74 212L183 211H293Z"></path><path stroke-width="0" id="E268-MJMAIN-5B" d="M118 -250V750H255V710H158V-210H255V-250H118Z"></path><path stroke-width="0" id="E268-MJMAIN-2212" d="M84 237T84 250T98 270H679Q694 262 694 250T679 230H98Q84 237 84 250Z"></path><path stroke-width="0" id="E268-MJMAIN-5D" d="M22 710V750H159V-250H22V-210H119V710H22Z"></path><path stroke-width="0" id="E268-MJMAIN-35" d="M164 157Q164 133 148 117T109 101H102Q148 22 224 22Q294 22 326 82Q345 115 345 210Q345 313 318 349Q292 382 260 382H254Q176 382 136 314Q132 307 129 306T114 304Q97 304 95 310Q93 314 93 485V614Q93 664 98 664Q100 666 102 666Q103 666 123 658T178 642T253 634Q324 634 389 662Q397 666 402 666Q410 666 410 648V635Q328 538 205 538Q174 538 149 544L139 546V374Q158 388 169 396T205 412T256 420Q337 420 393 355T449 201Q449 109 385 44T229 -22Q148 -22 99 32T50 154Q50 178 61 192T84 210T107 214Q132 214 148 197T164 157Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><g transform="translate(5074,-2469)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">算</text><g transform="translate(1052,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">法</text></g><use transform="scale(1.2)" xlink:href="#E268-MJMAINB-36" x="1963" y="0"></use><use transform="scale(1.2)" xlink:href="#E268-MJMAINB-2D" x="2538" y="0"></use><use transform="scale(1.2)" xlink:href="#E268-MJMAINB-33" x="2921" y="0"></use><g transform="translate(4945,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">半</text></g><g transform="translate(5998,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">梯</text></g><g transform="translate(7050,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">度</text></g><g transform="translate(8103,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">下</text></g><g transform="translate(9156,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">降</text></g><g transform="translate(10209,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">算</text></g><g transform="translate(11262,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">法</text></g><g transform="translate(12315,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">估</text></g><g transform="translate(13368,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">计</text></g><g transform="translate(14420,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">动</text></g><g transform="translate(15437,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">作</text></g><g transform="translate(16490,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">价</text></g><g transform="translate(17542,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">值</text></g><g transform="translate(18595,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">或</text></g><use transform="scale(1.2)" xlink:href="#E268-MJMAINB-53" x="16582" y="0"></use><use transform="scale(1.2)" xlink:href="#E268-MJMAINB-41" x="17221" y="0"></use><use transform="scale(1.2)" xlink:href="#E268-MJMAINB-52" x="18090" y="0"></use><use transform="scale(1.2)" xlink:href="#E268-MJMAINB-53" x="18952" y="0"></use><use transform="scale(1.2)" xlink:href="#E268-MJMAINB-41" x="19591" y="0"></use><g transform="translate(24802,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">算</text></g><g transform="translate(25855,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">法</text></g><g transform="translate(26907,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">求</text></g><g transform="translate(27960,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">最</text></g><g transform="translate(29013,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">优</text></g><g transform="translate(30066,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">策</text></g><g transform="translate(31119,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">略</text></g></g><g transform="translate(0,-10626)"><g transform="translate(-19,0)"><g transform="translate(0,6759)"><g><rect fill="black" stroke="none" width="1569" height="100" x="0" y="500"></rect></g></g><g transform="translate(0,-6560)"><g><rect fill="black" stroke="none" width="1569" height="100" x="0" y="-500"></rect></g></g></g><g transform="translate(1551,0)"><g transform="translate(0,6759)"><g><rect fill="black" stroke="none" width="41597" height="100" x="0" y="500"></rect></g></g><g transform="translate(0,5459)"><use xlink:href="#E268-MJMAIN-31"></use><use xlink:href="#E268-MJMAIN-2E" x="500" y="0"></use><g transform="translate(778,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(1608,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">初</text></g><g transform="translate(2439,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">始</text></g><g transform="translate(3269,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">化</text></g><g transform="translate(4100,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(4931,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">任</text></g><g transform="translate(5761,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">意</text></g><g transform="translate(6592,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">初</text></g><g transform="translate(7423,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">始</text></g><g transform="translate(8253,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">化</text></g><g transform="translate(9084,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">参</text></g><g transform="translate(9915,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">数</text></g><use xlink:href="#E268-MJMAINB-77" x="10995" y="0"></use><g transform="translate(11826,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">。</text></g></g></g><g transform="translate(0,4159)"><use xlink:href="#E268-MJMAIN-32"></use><use xlink:href="#E268-MJMAIN-2E" x="500" y="0"></use><g transform="translate(778,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(1608,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">时</text></g><g transform="translate(2439,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">序</text></g><g transform="translate(3269,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">差</text></g><g transform="translate(4100,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">分</text></g><g transform="translate(4931,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">更</text></g><g transform="translate(5761,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">新</text></g><g transform="translate(6592,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(7423,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">对</text></g><g transform="translate(8253,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">于</text></g><g transform="translate(9084,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">每</text></g><g transform="translate(9915,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">个</text></g><g transform="translate(10745,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">回</text></g><g transform="translate(11576,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">合</text></g><g transform="translate(12407,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">执</text></g><g transform="translate(13237,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">行</text></g><g transform="translate(14068,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">以</text></g><g transform="translate(14899,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">下</text></g><g transform="translate(15729,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">操</text></g><g transform="translate(16560,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">作</text></g><g transform="translate(17391,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">：</text></g></g><g transform="translate(0,2859)"><g transform="translate(2000,0)"><use xlink:href="#E268-MJMAIN-32"></use><use xlink:href="#E268-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E268-MJMAIN-31" x="778" y="0"></use><g transform="translate(1278,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(2108,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">初</text></g><g transform="translate(2939,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">始</text></g><g transform="translate(3769,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">化</text></g><g transform="translate(4600,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">状</text></g><g transform="translate(5431,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">态</text></g><g transform="translate(6261,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">动</text></g><g transform="translate(7092,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">作</text></g><g transform="translate(7923,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">对</text></g><g transform="translate(8753,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(9584,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">选</text></g><g transform="translate(10415,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">择</text></g><g transform="translate(11245,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">状</text></g><g transform="translate(12076,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">态</text></g><use xlink:href="#E268-MJMATHI-53" x="13157" y="0"></use><g transform="translate(13802,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">，</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">用</text></g></g><g transform="translate(15963,0)"><use xlink:href="#E268-MJMATHI-3C0" x="0" y="0"></use><use xlink:href="#E268-MJMAIN-28" x="573" y="0"></use><use xlink:href="#E268-MJMAIN-22C5" x="962" y="0"></use><use xlink:href="#E268-MJMAIN-2223" x="1517" y="0"></use><use xlink:href="#E268-MJMATHI-53" x="2073" y="0"></use><use xlink:href="#E268-MJMAIN-29" x="2718" y="0"></use></g><g transform="translate(19070,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">或</text></g></g><g transform="translate(20401,0)"><use xlink:href="#E268-MJMATHI-71" x="0" y="0"></use><use xlink:href="#E268-MJMAIN-28" x="460" y="0"></use><use xlink:href="#E268-MJMATHI-53" x="849" y="0"></use><use xlink:href="#E268-MJMAIN-2C" x="1494" y="0"></use><use xlink:href="#E268-MJMAIN-22C5" x="1938" y="0"></use><use xlink:href="#E268-MJMAIN-3B" x="2216" y="0"></use><use xlink:href="#E268-MJMAINB-77" x="2661" y="0"></use><use xlink:href="#E268-MJMAIN-29" x="3492" y="0"></use></g><g transform="translate(24282,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">确</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">定</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">动</text></g><g transform="translate(2741,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">作</text></g></g><use xlink:href="#E268-MJMATHI-41" x="28105" y="0"></use><g transform="translate(28855,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">。</text></g></g></g></g><g transform="translate(0,1509)"><g transform="translate(2000,0)"><use xlink:href="#E268-MJMAIN-32"></use><use xlink:href="#E268-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E268-MJMAIN-32" x="778" y="0"></use><g transform="translate(1972,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">若</text><g transform="translate(830,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">回</text></g><g transform="translate(1661,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">合</text></g><g transform="translate(2491,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">未</text></g><g transform="translate(3322,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">结</text></g><g transform="translate(4153,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">束</text></g><g transform="translate(4983,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">，</text></g><g transform="translate(5814,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">执</text></g><g transform="translate(6645,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">行</text></g><g transform="translate(7475,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">以</text></g><g transform="translate(8306,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">下</text></g><g transform="translate(9137,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">操</text></g><g transform="translate(9967,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">作</text></g><g transform="translate(10798,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">：</text></g></g></g></g><g transform="translate(0,200)"><g transform="translate(4000,0)"><use xlink:href="#E268-MJMAIN-32"></use><use xlink:href="#E268-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E268-MJMAIN-32" x="778" y="0"></use><use xlink:href="#E268-MJMAIN-2E" x="1278" y="0"></use><use xlink:href="#E268-MJMAIN-31" x="1556" y="0"></use><g transform="translate(2056,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(2886,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">采</text></g><g transform="translate(3717,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">样</text></g><g transform="translate(4547,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(5378,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">执</text></g><g transform="translate(6209,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">行</text></g><g transform="translate(7039,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">动</text></g><g transform="translate(7870,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">作</text></g><use xlink:href="#E268-MJMATHI-41" x="8951" y="0"></use><g transform="translate(9701,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">，</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">观</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">测</text></g><g transform="translate(2741,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">得</text></g><g transform="translate(3572,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">到</text></g><g transform="translate(4403,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">的</text></g><g transform="translate(5233,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">奖</text></g><g transform="translate(6064,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">励</text></g></g><use xlink:href="#E268-MJMATHI-52" x="16846" y="0"></use><g transform="translate(17605,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">和</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">新</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">状</text></g><g transform="translate(2741,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">态</text></g></g><g transform="translate(21428,0)"><use xlink:href="#E268-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E268-MJMAIN-2032" x="925" y="583"></use></g><g transform="translate(22377,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">；</text></g></g></g></g><g transform="translate(0,-1109)"><g transform="translate(4000,0)"><use xlink:href="#E268-MJMAIN-32"></use><use xlink:href="#E268-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E268-MJMAIN-32" x="778" y="0"></use><use xlink:href="#E268-MJMAIN-2E" x="1278" y="0"></use><use xlink:href="#E268-MJMAIN-32" x="1556" y="0"></use><g transform="translate(2750,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">用</text></g><g transform="translate(3831,0)"><use xlink:href="#E268-MJMATHI-3C0" x="0" y="0"></use><use xlink:href="#E268-MJMAIN-28" x="573" y="0"></use><use xlink:href="#E268-MJMAIN-22C5" x="962" y="0"></use><use xlink:href="#E268-MJMAIN-2223" x="1517" y="0"></use><use xlink:href="#E268-MJMATHI-53" x="2073" y="0"></use><use xlink:href="#E268-MJMAIN-29" x="2718" y="0"></use></g><g transform="translate(6938,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">或</text></g></g><g transform="translate(8269,0)"><use xlink:href="#E268-MJMATHI-71" x="0" y="0"></use><use xlink:href="#E268-MJMAIN-28" x="460" y="0"></use><use xlink:href="#E268-MJMATHI-53" x="849" y="0"></use><use xlink:href="#E268-MJMAIN-2C" x="1494" y="0"></use><use xlink:href="#E268-MJMAIN-22C5" x="1938" y="0"></use><use xlink:href="#E268-MJMAIN-3B" x="2216" y="0"></use><use xlink:href="#E268-MJMAINB-77" x="2661" y="0"></use><use xlink:href="#E268-MJMAIN-29" x="3492" y="0"></use></g><g transform="translate(12150,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">确</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">定</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">动</text></g><g transform="translate(2741,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">作</text></g></g><g transform="translate(15973,0)"><use xlink:href="#E268-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E268-MJMAIN-2032" x="1060" y="583"></use></g><g transform="translate(17017,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">；</text></g></g></g></g><g transform="translate(0,-2467)"><g transform="translate(4000,0)"><use xlink:href="#E268-MJMAIN-32"></use><use xlink:href="#E268-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E268-MJMAIN-32" x="778" y="0"></use><use xlink:href="#E268-MJMAIN-2E" x="1278" y="0"></use><use xlink:href="#E268-MJMAIN-33" x="1556" y="0"></use><g transform="translate(2056,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(2886,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">计</text></g><g transform="translate(3717,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">算</text></g><g transform="translate(4547,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">回</text></g><g transform="translate(5378,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">报</text></g><g transform="translate(6209,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">的</text></g><g transform="translate(7039,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">估</text></g><g transform="translate(7870,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">计</text></g><g transform="translate(8701,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">值</text></g><g transform="translate(9531,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(10362,0)"><use xlink:href="#E268-MJMATHI-55" x="0" y="0"></use><use xlink:href="#E268-MJMAIN-2190" x="1044" y="0"></use><use xlink:href="#E268-MJMATHI-52" x="2322" y="0"></use><use xlink:href="#E268-MJMAIN-2B" x="3303" y="0"></use><use xlink:href="#E268-MJMATHI-3B3" x="4304" y="0"></use><use xlink:href="#E268-MJMATHI-71" x="4847" y="0"></use><use xlink:href="#E268-MJMAIN-28" x="5307" y="0"></use><g transform="translate(5696,0)"><use xlink:href="#E268-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E268-MJMAIN-2032" x="925" y="583"></use></g><use xlink:href="#E268-MJMAIN-2C" x="6645" y="0"></use><g transform="translate(7089,0)"><use xlink:href="#E268-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E268-MJMAIN-2032" x="1060" y="583"></use></g><use xlink:href="#E268-MJMAIN-3B" x="8134" y="0"></use><use xlink:href="#E268-MJMAINB-77" x="8578" y="0"></use><use xlink:href="#E268-MJMAIN-29" x="9409" y="0"></use></g><g transform="translate(20161,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">；</text></g></g></g></g><g transform="translate(0,-3901)"><g transform="translate(4000,0)"><use xlink:href="#E268-MJMAIN-32"></use><use xlink:href="#E268-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E268-MJMAIN-32" x="778" y="0"></use><use xlink:href="#E268-MJMAIN-2E" x="1278" y="0"></use><use xlink:href="#E268-MJMAIN-34" x="1556" y="0"></use><g transform="translate(2056,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(2886,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">更</text></g><g transform="translate(3717,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">新</text></g><g transform="translate(4547,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">价</text></g><g transform="translate(5378,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">值</text></g><g transform="translate(6209,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(7039,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">更</text></g><g transform="translate(7870,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">新</text></g><use xlink:href="#E268-MJMAINB-77" x="8951" y="0"></use><g transform="translate(9782,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">以</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">减</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">小</text></g></g><g transform="translate(12774,0)"><use xlink:href="#E268-MJMAIN-5B" x="0" y="0"></use><use xlink:href="#E268-MJMATHI-55" x="278" y="0"></use><use xlink:href="#E268-MJMAIN-2212" x="1267" y="0"></use><use xlink:href="#E268-MJMATHI-71" x="2267" y="0"></use><use xlink:href="#E268-MJMAIN-28" x="2727" y="0"></use><use xlink:href="#E268-MJMATHI-53" x="3116" y="0"></use><use xlink:href="#E268-MJMAIN-2C" x="3761" y="0"></use><use xlink:href="#E268-MJMATHI-41" x="4206" y="0"></use><use xlink:href="#E268-MJMAIN-3B" x="4956" y="0"></use><use xlink:href="#E268-MJMAINB-77" x="5400" y="0"></use><use xlink:href="#E268-MJMAIN-29" x="6231" y="0"></use><g transform="translate(6620,0)"><use xlink:href="#E268-MJMAIN-5D" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E268-MJMAIN-32" x="393" y="583"></use></g></g><g transform="translate(20126,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">；</text></g></g></g></g><g transform="translate(0,-5260)"><g transform="translate(4000,0)"><use xlink:href="#E268-MJMAIN-32"></use><use xlink:href="#E268-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E268-MJMAIN-32" x="778" y="0"></use><use xlink:href="#E268-MJMAIN-2E" x="1278" y="0"></use><use xlink:href="#E268-MJMAIN-35" x="1556" y="0"></use><g transform="translate(2306,0)"><use xlink:href="#E268-MJMATHI-53" x="444" y="0"></use><use xlink:href="#E268-MJMAIN-2190" x="1367" y="0"></use><g transform="translate(2645,0)"><use xlink:href="#E268-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E268-MJMAIN-2032" x="925" y="583"></use></g><use xlink:href="#E268-MJMAIN-2C" x="3594" y="0"></use><use xlink:href="#E268-MJMATHI-41" x="4316" y="0"></use><use xlink:href="#E268-MJMAIN-2190" x="5344" y="0"></use><g transform="translate(6622,0)"><use xlink:href="#E268-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E268-MJMAIN-2032" x="1060" y="583"></use></g></g><g transform="translate(9972,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">。</text></g></g></g></g><g transform="translate(0,-6560)"><g><rect fill="black" stroke="none" width="41597" height="100" x="0" y="-500"></rect></g></g></g></g></g></svg></span></div><script type="math/tex; mode=display" id="MathJax-Element-241">\; \\ \; \\
\large \textbf{算法 6-3   半梯度下降算法估计动作价值或 SARSA 算法求最优策略} \\
\begin{split}
\rule[5pt]{10mm}{0.1em} &\rule[5pt]{265mm}{0.1em} \\
&\text{1.（初始化）任意初始化参数 $\bold w$ 。} \\
&\text{2.（时序差分更新）对于每个回合执行以下操作：} \\
&\qquad \text{2.1（初始化状态动作对）选择状态 $S$ ，用 $\pi(\cdot \mid S)$ 或 $q(S,\cdot;\bold w)$ 确定动作 $A$ 。} \\
&\qquad \text{2.2 $\;\,$若回合未结束，执行以下操作：} \\
&\qquad \qquad \text{2.2.1（采样）执行动作 $A$ ，观测得到的奖励 $R$ 和新状态 $S'$ ；} \\
&\qquad \qquad \text{2.2.2 $\;\,$用 $\pi(\cdot \mid S)$ 或 $q(S,\cdot;\bold w)$ 确定动作 $A'$ ；} \\
&\qquad \qquad \text{2.2.3（计算回报的估计值）$U \leftarrow R + \gamma q(S',A';\bold w)$ ；} \\
&\qquad \qquad \text{2.2.4（更新价值）更新 $\bold w$ 以减小 $[U-q(S,A;\bold w)]^2$ ；} \\
&\qquad \qquad \text{2.2.5 $\;\, S \leftarrow S',\; A \leftarrow A'$ 。}\\
\rule[-5pt]{10mm}{0.1em} &\rule[-5pt]{265mm}{0.1em}
\end{split}
\; \\ \; \\</script></div></div><div contenteditable="false" spellcheck="false" class="mathjax-block md-end-block md-math-block md-rawblock" id="mathjax-n16" cid="n16" mdtype="math_block"><div class="md-rawblock-container md-math-container" tabindex="-1"><div class="MathJax_SVG_Display"><span class="MathJax_SVG" id="MathJax-Element-242-Frame" tabindex="-1" style="font-size: 100%; display: inline-block; zoom: 0.971345;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="101.294ex" height="58.273ex" viewBox="-18.1 -43.5 43612.5 25089.7" role="img" focusable="false" style="vertical-align: -58.172ex; margin-left: -0.042ex; max-width: 100%;"><defs><path stroke-width="0" id="E269-MJMAINB-36" d="M48 318Q48 395 68 456T120 553T193 613T273 646T350 655Q425 655 461 616T497 524Q497 485 475 468T428 451Q399 451 378 470T357 521Q357 565 403 588Q375 601 351 601Q313 601 282 584Q242 565 222 526Q199 473 199 367Q201 369 210 380T227 396T246 410T275 422T312 426Q438 426 494 332Q526 285 526 208V199Q526 112 465 53Q428 17 388 3T285 -11Q236 -11 195 7T135 43T104 80Q48 165 48 318ZM375 231V244V268Q375 295 373 310T364 342T341 366T299 374H297Q231 374 208 287Q200 257 200 196Q201 120 209 100Q231 47 288 47Q351 47 368 90Q375 112 375 231Z"></path><path stroke-width="0" id="E269-MJMAINB-2D" d="M13 166V278H318V166H13Z"></path><path stroke-width="0" id="E269-MJMAINB-34" d="M531 0Q510 3 381 3Q238 3 214 0H201V62H313V155H32V217L205 434Q342 606 362 630T387 655L391 656Q395 656 401 656T414 656H427Q447 656 451 645Q453 641 453 429V217H542V155H453V62H542V0H531ZM324 217V494L103 218L213 217H324Z"></path><path stroke-width="0" id="E269-MJMAINB-53" d="M64 493Q64 582 120 636T264 696H272Q280 697 285 697Q380 697 454 645L480 669Q484 672 488 676T495 683T500 688T504 691T508 693T511 695T514 696T517 697T522 697Q536 697 539 691T542 652V577Q542 557 542 532T543 500Q543 472 540 465T524 458H511H505Q489 458 485 461T479 478Q472 529 449 564T393 614T336 634T287 639Q228 639 203 610T177 544Q177 517 195 493T247 457Q253 454 343 436T475 391Q574 326 574 207V200Q574 163 559 120Q517 12 389 -9Q380 -10 346 -10Q308 -10 275 -5T221 7T184 22T160 35T151 40L126 17Q122 14 118 10T111 3T106 -2T102 -5T98 -7T95 -9T92 -10T89 -11T84 -11Q70 -11 67 -4T64 35V108Q64 128 64 153T63 185Q63 203 63 211T69 223T77 227T94 228H100Q118 228 122 225T126 205Q130 125 193 88T345 51Q408 51 434 82T460 157Q460 196 439 221T388 257Q384 259 305 276T221 295Q155 313 110 366T64 493Z"></path><path stroke-width="0" id="E269-MJMAINB-41" d="M296 0Q278 3 164 3Q58 3 49 0H40V62H92Q144 62 144 64Q388 682 397 689Q403 698 434 698Q463 698 471 689Q475 686 538 530T663 218L724 64Q724 62 776 62H828V0H817Q796 3 658 3Q509 3 485 0H472V62H517Q561 62 561 63L517 175H262L240 120Q218 65 217 64Q217 62 261 62H306V0H296ZM390 237L492 238L440 365Q390 491 388 491Q287 239 287 237H390Z"></path><path stroke-width="0" id="E269-MJMAINB-52" d="M394 0Q370 3 222 3Q75 3 51 0H39V62H147V624H39V686H234Q256 686 299 686T362 687Q479 687 554 669T681 593Q716 550 716 497Q716 390 568 338Q569 337 572 336T577 332Q605 317 623 300T650 258T662 218T668 172Q678 98 689 76Q707 40 748 40Q770 40 780 54T795 88T801 111Q805 117 827 117H831Q846 117 852 113T858 92Q857 78 852 63T834 30T797 1T739 -11Q630 -11 580 12T511 87Q506 104 506 168Q506 170 506 178T507 194Q507 289 438 313Q424 318 356 318H298V62H406V0H394ZM366 369Q459 370 490 381Q548 402 548 476V498V517Q548 578 513 600Q479 624 392 624H358H298V369H366Z"></path><path stroke-width="0" id="E269-MJMAINB-51" d="M64 339Q64 431 96 502T182 614T295 675T420 696Q469 696 481 695Q620 680 709 589T798 339Q798 255 768 184Q720 77 611 26L600 21Q635 -26 682 -26H696Q769 -26 769 0Q769 7 774 12T787 18Q805 18 805 -7V-13Q803 -64 785 -106T737 -171Q720 -183 697 -191Q687 -193 668 -193Q636 -193 613 -182T575 -144T552 -94T532 -27Q531 -23 530 -16T528 -6T526 -3L512 -5Q499 -7 477 -8T431 -10Q393 -10 382 -9Q238 8 151 97T64 339ZM326 80Q326 113 356 138T430 163Q492 163 542 100L553 86Q554 85 561 91T578 108Q637 179 637 330Q637 430 619 498T548 604Q500 641 425 641Q408 641 390 637T347 623T299 590T259 535Q226 469 226 338Q226 244 246 180T318 79L325 74Q326 74 326 80ZM506 58Q480 112 433 112Q412 112 395 104T378 77Q378 44 431 44Q480 44 506 58Z"></path><path stroke-width="0" id="E269-MJMAIN-31" d="M213 578L200 573Q186 568 160 563T102 556H83V602H102Q149 604 189 617T245 641T273 663Q275 666 285 666Q294 666 302 660V361L303 61Q310 54 315 52T339 48T401 46H427V0H416Q395 3 257 3Q121 3 100 0H88V46H114Q136 46 152 46T177 47T193 50T201 52T207 57T213 61V578Z"></path><path stroke-width="0" id="E269-MJMAIN-2E" d="M78 60Q78 84 95 102T138 120Q162 120 180 104T199 61Q199 36 182 18T139 0T96 17T78 60Z"></path><path stroke-width="0" id="E269-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E269-MJMAIN-32" d="M109 429Q82 429 66 447T50 491Q50 562 103 614T235 666Q326 666 387 610T449 465Q449 422 429 383T381 315T301 241Q265 210 201 149L142 93L218 92Q375 92 385 97Q392 99 409 186V189H449V186Q448 183 436 95T421 3V0H50V19V31Q50 38 56 46T86 81Q115 113 136 137Q145 147 170 174T204 211T233 244T261 278T284 308T305 340T320 369T333 401T340 431T343 464Q343 527 309 573T212 619Q179 619 154 602T119 569T109 550Q109 549 114 549Q132 549 151 535T170 489Q170 464 154 447T109 429Z"></path><path stroke-width="0" id="E269-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E269-MJMATHI-3C0" d="M132 -11Q98 -11 98 22V33L111 61Q186 219 220 334L228 358H196Q158 358 142 355T103 336Q92 329 81 318T62 297T53 285Q51 284 38 284Q19 284 19 294Q19 300 38 329T93 391T164 429Q171 431 389 431Q549 431 553 430Q573 423 573 402Q573 371 541 360Q535 358 472 358H408L405 341Q393 269 393 222Q393 170 402 129T421 65T431 37Q431 20 417 5T381 -10Q370 -10 363 -7T347 17T331 77Q330 86 330 121Q330 170 339 226T357 318T367 358H269L268 354Q268 351 249 275T206 114T175 17Q164 -11 132 -11Z"></path><path stroke-width="0" id="E269-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E269-MJMAIN-22C5" d="M78 250Q78 274 95 292T138 310Q162 310 180 294T199 251Q199 226 182 208T139 190T96 207T78 250Z"></path><path stroke-width="0" id="E269-MJMAIN-2223" d="M139 -249H137Q125 -249 119 -235V251L120 737Q130 750 139 750Q152 750 159 735V-235Q151 -249 141 -249H139Z"></path><path stroke-width="0" id="E269-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E269-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E269-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E269-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E269-MJMATHI-41" d="M208 74Q208 50 254 46Q272 46 272 35Q272 34 270 22Q267 8 264 4T251 0Q249 0 239 0T205 1T141 2Q70 2 50 0H42Q35 7 35 11Q37 38 48 46H62Q132 49 164 96Q170 102 345 401T523 704Q530 716 547 716H555H572Q578 707 578 706L606 383Q634 60 636 57Q641 46 701 46Q726 46 726 36Q726 34 723 22Q720 7 718 4T704 0Q701 0 690 0T651 1T578 2Q484 2 455 0H443Q437 6 437 9T439 27Q443 40 445 43L449 46H469Q523 49 533 63L521 213H283L249 155Q208 86 208 74ZM516 260Q516 271 504 416T490 562L463 519Q447 492 400 412L310 260L413 259Q516 259 516 260Z"></path><path stroke-width="0" id="E269-MJMATHI-52" d="M230 637Q203 637 198 638T193 649Q193 676 204 682Q206 683 378 683Q550 682 564 680Q620 672 658 652T712 606T733 563T739 529Q739 484 710 445T643 385T576 351T538 338L545 333Q612 295 612 223Q612 212 607 162T602 80V71Q602 53 603 43T614 25T640 16Q668 16 686 38T712 85Q717 99 720 102T735 105Q755 105 755 93Q755 75 731 36Q693 -21 641 -21H632Q571 -21 531 4T487 82Q487 109 502 166T517 239Q517 290 474 313Q459 320 449 321T378 323H309L277 193Q244 61 244 59Q244 55 245 54T252 50T269 48T302 46H333Q339 38 339 37T336 19Q332 6 326 0H311Q275 2 180 2Q146 2 117 2T71 2T50 1Q33 1 33 10Q33 12 36 24Q41 43 46 45Q50 46 61 46H67Q94 46 127 49Q141 52 146 61Q149 65 218 339T287 628Q287 635 230 637ZM630 554Q630 586 609 608T523 636Q521 636 500 636T462 637H440Q393 637 386 627Q385 624 352 494T319 361Q319 360 388 360Q466 361 492 367Q556 377 592 426Q608 449 619 486T630 554Z"></path><path stroke-width="0" id="E269-MJMAIN-2032" d="M79 43Q73 43 52 49T30 61Q30 68 85 293T146 528Q161 560 198 560Q218 560 240 545T262 501Q262 496 260 486Q259 479 173 263T84 45T79 43Z"></path><path stroke-width="0" id="E269-MJMAIN-33" d="M127 463Q100 463 85 480T69 524Q69 579 117 622T233 665Q268 665 277 664Q351 652 390 611T430 522Q430 470 396 421T302 350L299 348Q299 347 308 345T337 336T375 315Q457 262 457 175Q457 96 395 37T238 -22Q158 -22 100 21T42 130Q42 158 60 175T105 193Q133 193 151 175T169 130Q169 119 166 110T159 94T148 82T136 74T126 70T118 67L114 66Q165 21 238 21Q293 21 321 74Q338 107 338 175V195Q338 290 274 322Q259 328 213 329L171 330L168 332Q166 335 166 348Q166 366 174 366Q202 366 232 371Q266 376 294 413T322 525V533Q322 590 287 612Q265 626 240 626Q208 626 181 615T143 592T132 580H135Q138 579 143 578T153 573T165 566T175 555T183 540T186 520Q186 498 172 481T127 463Z"></path><path stroke-width="0" id="E269-MJMATHI-55" d="M107 637Q73 637 71 641Q70 643 70 649Q70 673 81 682Q83 683 98 683Q139 681 234 681Q268 681 297 681T342 682T362 682Q378 682 378 672Q378 670 376 658Q371 641 366 638H364Q362 638 359 638T352 638T343 637T334 637Q295 636 284 634T266 623Q265 621 238 518T184 302T154 169Q152 155 152 140Q152 86 183 55T269 24Q336 24 403 69T501 205L552 406Q599 598 599 606Q599 633 535 637Q511 637 511 648Q511 650 513 660Q517 676 519 679T529 683Q532 683 561 682T645 680Q696 680 723 681T752 682Q767 682 767 672Q767 650 759 642Q756 637 737 637Q666 633 648 597Q646 592 598 404Q557 235 548 205Q515 105 433 42T263 -22Q171 -22 116 34T60 167V183Q60 201 115 421Q164 622 164 628Q164 635 107 637Z"></path><path stroke-width="0" id="E269-MJMAIN-2190" d="M944 261T944 250T929 230H165Q167 228 182 216T211 189T244 152T277 96T303 25Q308 7 308 0Q308 -11 288 -11Q281 -11 278 -11T272 -7T267 2T263 21Q245 94 195 151T73 236Q58 242 55 247Q55 254 59 257T73 264Q121 283 158 314T215 375T247 434T264 480L267 497Q269 503 270 505T275 509T288 511Q308 511 308 500Q308 493 303 475Q293 438 278 406T246 352T215 315T185 287T165 270H929Q944 261 944 250Z"></path><path stroke-width="0" id="E269-MJMAIN-2B" d="M56 237T56 250T70 270H369V420L370 570Q380 583 389 583Q402 583 409 568V270H707Q722 262 722 250T707 230H409V-68Q401 -82 391 -82H389H387Q375 -82 369 -68V230H70Q56 237 56 250Z"></path><path stroke-width="0" id="E269-MJMATHI-3B3" d="M31 249Q11 249 11 258Q11 275 26 304T66 365T129 418T206 441Q233 441 239 440Q287 429 318 386T371 255Q385 195 385 170Q385 166 386 166L398 193Q418 244 443 300T486 391T508 430Q510 431 524 431H537Q543 425 543 422Q543 418 522 378T463 251T391 71Q385 55 378 6T357 -100Q341 -165 330 -190T303 -216Q286 -216 286 -188Q286 -138 340 32L346 51L347 69Q348 79 348 100Q348 257 291 317Q251 355 196 355Q148 355 108 329T51 260Q49 251 47 251Q45 249 31 249Z"></path><path stroke-width="0" id="E269-MJMATHI-76" d="M173 380Q173 405 154 405Q130 405 104 376T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Q21 294 29 316T53 368T97 419T160 441Q202 441 225 417T249 361Q249 344 246 335Q246 329 231 291T200 202T182 113Q182 86 187 69Q200 26 250 26Q287 26 319 60T369 139T398 222T409 277Q409 300 401 317T383 343T365 361T357 383Q357 405 376 424T417 443Q436 443 451 425T467 367Q467 340 455 284T418 159T347 40T241 -11Q177 -11 139 22Q102 54 102 117Q102 148 110 181T151 298Q173 362 173 380Z"></path><path stroke-width="0" id="E269-MJMAIN-53" d="M55 507Q55 590 112 647T243 704H257Q342 704 405 641L426 672Q431 679 436 687T446 700L449 704Q450 704 453 704T459 705H463Q466 705 472 699V462L466 456H448Q437 456 435 459T430 479Q413 605 329 646Q292 662 254 662Q201 662 168 626T135 542Q135 508 152 480T200 435Q210 431 286 412T370 389Q427 367 463 314T500 191Q500 110 448 45T301 -21Q245 -21 201 -4T140 27L122 41Q118 36 107 21T87 -7T78 -21Q76 -22 68 -22H64Q61 -22 55 -16V101Q55 220 56 222Q58 227 76 227H89Q95 221 95 214Q95 182 105 151T139 90T205 42T305 24Q352 24 386 62T420 155Q420 198 398 233T340 281Q284 295 266 300Q261 301 239 306T206 314T174 325T141 343T112 367T85 402Q55 451 55 507Z"></path><path stroke-width="0" id="E269-MJMAIN-41" d="M255 0Q240 3 140 3Q48 3 39 0H32V46H47Q119 49 139 88Q140 91 192 245T295 553T348 708Q351 716 366 716H376Q396 715 400 709Q402 707 508 390L617 67Q624 54 636 51T687 46H717V0H708Q699 3 581 3Q458 3 437 0H427V46H440Q510 46 510 64Q510 66 486 138L462 209H229L209 150Q189 91 189 85Q189 72 209 59T259 46H264V0H255ZM447 255L345 557L244 256Q244 255 345 255H447Z"></path><path stroke-width="0" id="E269-MJMAIN-52" d="M130 622Q123 629 119 631T103 634T60 637H27V683H202H236H300Q376 683 417 677T500 648Q595 600 609 517Q610 512 610 501Q610 468 594 439T556 392T511 361T472 343L456 338Q459 335 467 332Q497 316 516 298T545 254T559 211T568 155T578 94Q588 46 602 31T640 16H645Q660 16 674 32T692 87Q692 98 696 101T712 105T728 103T732 90Q732 59 716 27T672 -16Q656 -22 630 -22Q481 -16 458 90Q456 101 456 163T449 246Q430 304 373 320L363 322L297 323H231V192L232 61Q238 51 249 49T301 46H334V0H323Q302 3 181 3Q59 3 38 0H27V46H60Q102 47 111 49T130 61V622ZM491 499V509Q491 527 490 539T481 570T462 601T424 623T362 636Q360 636 340 636T304 637H283Q238 637 234 628Q231 624 231 492V360H289Q390 360 434 378T489 456Q491 467 491 499Z"></path><path stroke-width="0" id="E269-MJSZ2-2211" d="M60 948Q63 950 665 950H1267L1325 815Q1384 677 1388 669H1348L1341 683Q1320 724 1285 761Q1235 809 1174 838T1033 881T882 898T699 902H574H543H251L259 891Q722 258 724 252Q725 250 724 246Q721 243 460 -56L196 -356Q196 -357 407 -357Q459 -357 548 -357T676 -358Q812 -358 896 -353T1063 -332T1204 -283T1307 -196Q1328 -170 1348 -124H1388Q1388 -125 1381 -145T1356 -210T1325 -294L1267 -449L666 -450Q64 -450 61 -448Q55 -446 55 -439Q55 -437 57 -433L590 177Q590 178 557 222T452 366T322 544L56 909L55 924Q55 945 60 948Z"></path><path stroke-width="0" id="E269-MJMATHI-61" d="M33 157Q33 258 109 349T280 441Q331 441 370 392Q386 422 416 422Q429 422 439 414T449 394Q449 381 412 234T374 68Q374 43 381 35T402 26Q411 27 422 35Q443 55 463 131Q469 151 473 152Q475 153 483 153H487Q506 153 506 144Q506 138 501 117T481 63T449 13Q436 0 417 -8Q409 -10 393 -10Q359 -10 336 5T306 36L300 51Q299 52 296 50Q294 48 292 46Q233 -10 172 -10Q117 -10 75 30T33 157ZM351 328Q351 334 346 350T323 385T277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q217 26 254 59T298 110Q300 114 325 217T351 328Z"></path><path stroke-width="0" id="E269-MJMATHI-3B5" d="M190 -22Q124 -22 76 11T27 107Q27 174 97 232L107 239L99 248Q76 273 76 304Q76 364 144 408T290 452H302Q360 452 405 421Q428 405 428 392Q428 381 417 369T391 356Q382 356 371 365T338 383T283 392Q217 392 167 368T116 308Q116 289 133 272Q142 263 145 262T157 264Q188 278 238 278H243Q308 278 308 247Q308 206 223 206Q177 206 142 219L132 212Q68 169 68 112Q68 39 201 39Q253 39 286 49T328 72T345 94T362 105Q376 103 376 88Q376 79 365 62T334 26T275 -8T190 -22Z"></path><path stroke-width="0" id="E269-MJMAIN-51" d="M56 341Q56 499 157 602T388 705Q521 705 621 601T722 341Q722 275 703 218T660 127T603 63T555 25T525 9Q524 8 524 8H523Q524 5 526 -1T537 -21T555 -47T581 -67T615 -76Q653 -76 678 -56T706 -3Q707 10 716 10Q721 10 728 5L727 -13Q727 -88 697 -140T606 -193Q563 -193 538 -166T498 -83Q483 -23 483 -8L471 -11Q459 -14 435 -18T388 -22Q254 -22 155 81T56 341ZM607 339Q607 429 586 496T531 598T461 649T390 665T318 649T248 598T192 496T170 339Q170 143 277 57Q301 39 305 39L304 42Q304 44 304 46Q301 53 301 68Q301 101 325 128T391 155Q454 155 495 70L501 58Q549 91 578 164Q607 234 607 339ZM385 18Q404 18 425 23T459 33T472 40Q471 47 468 57T449 88T412 115Q398 117 386 117Q367 117 353 102T338 67Q338 48 351 33T385 18Z"></path><path stroke-width="0" id="E269-MJMAIN-6D" d="M41 46H55Q94 46 102 60V68Q102 77 102 91T102 122T103 161T103 203Q103 234 103 269T102 328V351Q99 370 88 376T43 385H25V408Q25 431 27 431L37 432Q47 433 65 434T102 436Q119 437 138 438T167 441T178 442H181V402Q181 364 182 364T187 369T199 384T218 402T247 421T285 437Q305 442 336 442Q351 442 364 440T387 434T406 426T421 417T432 406T441 395T448 384T452 374T455 366L457 361L460 365Q463 369 466 373T475 384T488 397T503 410T523 422T546 432T572 439T603 442Q729 442 740 329Q741 322 741 190V104Q741 66 743 59T754 49Q775 46 803 46H819V0H811L788 1Q764 2 737 2T699 3Q596 3 587 0H579V46H595Q656 46 656 62Q657 64 657 200Q656 335 655 343Q649 371 635 385T611 402T585 404Q540 404 506 370Q479 343 472 315T464 232V168V108Q464 78 465 68T468 55T477 49Q498 46 526 46H542V0H534L510 1Q487 2 460 2T422 3Q319 3 310 0H302V46H318Q379 46 379 62Q380 64 380 200Q379 335 378 343Q372 371 358 385T334 402T308 404Q263 404 229 370Q202 343 195 315T187 232V168V108Q187 78 188 68T191 55T200 49Q221 46 249 46H265V0H257L234 1Q210 2 183 2T145 3Q42 3 33 0H25V46H41Z"></path><path stroke-width="0" id="E269-MJMAIN-61" d="M137 305T115 305T78 320T63 359Q63 394 97 421T218 448Q291 448 336 416T396 340Q401 326 401 309T402 194V124Q402 76 407 58T428 40Q443 40 448 56T453 109V145H493V106Q492 66 490 59Q481 29 455 12T400 -6T353 12T329 54V58L327 55Q325 52 322 49T314 40T302 29T287 17T269 6T247 -2T221 -8T190 -11Q130 -11 82 20T34 107Q34 128 41 147T68 188T116 225T194 253T304 268H318V290Q318 324 312 340Q290 411 215 411Q197 411 181 410T156 406T148 403Q170 388 170 359Q170 334 154 320ZM126 106Q126 75 150 51T209 26Q247 26 276 49T315 109Q317 116 318 175Q318 233 317 233Q309 233 296 232T251 223T193 203T147 166T126 106Z"></path><path stroke-width="0" id="E269-MJMAIN-78" d="M201 0Q189 3 102 3Q26 3 17 0H11V46H25Q48 47 67 52T96 61T121 78T139 96T160 122T180 150L226 210L168 288Q159 301 149 315T133 336T122 351T113 363T107 370T100 376T94 379T88 381T80 383Q74 383 44 385H16V431H23Q59 429 126 429Q219 429 229 431H237V385Q201 381 201 369Q201 367 211 353T239 315T268 274L272 270L297 304Q329 345 329 358Q329 364 327 369T322 376T317 380T310 384L307 385H302V431H309Q324 428 408 428Q487 428 493 431H499V385H492Q443 385 411 368Q394 360 377 341T312 257L296 236L358 151Q424 61 429 57T446 50Q464 46 499 46H516V0H510H502Q494 1 482 1T457 2T432 2T414 3Q403 3 377 3T327 1L304 0H295V46H298Q309 46 320 51T331 63Q331 65 291 120L250 175Q249 174 219 133T185 88Q181 83 181 74Q181 63 188 55T206 46Q208 46 208 23V0H201Z"></path><path stroke-width="0" id="E269-MJMAIN-34" d="M462 0Q444 3 333 3Q217 3 199 0H190V46H221Q241 46 248 46T265 48T279 53T286 61Q287 63 287 115V165H28V211L179 442Q332 674 334 675Q336 677 355 677H373L379 671V211H471V165H379V114Q379 73 379 66T385 54Q393 47 442 46H471V0H462ZM293 211V545L74 212L183 211H293Z"></path><path stroke-width="0" id="E269-MJMAIN-5B" d="M118 -250V750H255V710H158V-210H255V-250H118Z"></path><path stroke-width="0" id="E269-MJMAIN-2212" d="M84 237T84 250T98 270H679Q694 262 694 250T679 230H98Q84 237 84 250Z"></path><path stroke-width="0" id="E269-MJMAIN-5D" d="M22 710V750H159V-250H22V-210H119V710H22Z"></path><path stroke-width="0" id="E269-MJMAIN-35" d="M164 157Q164 133 148 117T109 101H102Q148 22 224 22Q294 22 326 82Q345 115 345 210Q345 313 318 349Q292 382 260 382H254Q176 382 136 314Q132 307 129 306T114 304Q97 304 95 310Q93 314 93 485V614Q93 664 98 664Q100 666 102 666Q103 666 123 658T178 642T253 634Q324 634 389 662Q397 666 402 666Q410 666 410 648V635Q328 538 205 538Q174 538 149 544L139 546V374Q158 388 169 396T205 412T256 420Q337 420 393 355T449 201Q449 109 385 44T229 -22Q148 -22 99 32T50 154Q50 178 61 192T84 210T107 214Q132 214 148 197T164 157Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><g transform="translate(4306,-2469)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">算</text><g transform="translate(1052,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">法</text></g><use transform="scale(1.2)" xlink:href="#E269-MJMAINB-36" x="1963" y="0"></use><use transform="scale(1.2)" xlink:href="#E269-MJMAINB-2D" x="2538" y="0"></use><use transform="scale(1.2)" xlink:href="#E269-MJMAINB-34" x="2921" y="0"></use><g transform="translate(4945,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">半</text></g><g transform="translate(5998,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">梯</text></g><g transform="translate(7050,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">度</text></g><g transform="translate(8103,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">下</text></g><g transform="translate(9156,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">降</text></g><g transform="translate(10209,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">算</text></g><g transform="translate(11262,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">法</text></g><g transform="translate(12315,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">估</text></g><g transform="translate(13368,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">计</text></g><g transform="translate(14420,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">状</text></g><g transform="translate(15473,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">态</text></g><g transform="translate(16526,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">价</text></g><g transform="translate(17579,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">值</text></g><g transform="translate(18632,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">或</text></g><g transform="translate(19685,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">期</text></g><g transform="translate(20738,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">望</text></g><use transform="scale(1.2)" xlink:href="#E269-MJMAINB-53" x="18367" y="0"></use><use transform="scale(1.2)" xlink:href="#E269-MJMAINB-41" x="19006" y="0"></use><use transform="scale(1.2)" xlink:href="#E269-MJMAINB-52" x="19875" y="0"></use><use transform="scale(1.2)" xlink:href="#E269-MJMAINB-53" x="20737" y="0"></use><use transform="scale(1.2)" xlink:href="#E269-MJMAINB-41" x="21376" y="0"></use><g transform="translate(26944,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">算</text></g><g transform="translate(27997,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">法</text></g><g transform="translate(29050,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">或</text></g><use transform="scale(1.2)" xlink:href="#E269-MJMAINB-51" x="25294" y="0"></use><g transform="translate(31639,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">学</text></g><g transform="translate(32692,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">习</text></g></g><g transform="translate(0,-13495)"><g transform="translate(-19,0)"><g transform="translate(0,9564)"><g><rect fill="black" stroke="none" width="1569" height="100" x="0" y="500"></rect></g></g><g transform="translate(0,-9365)"><g><rect fill="black" stroke="none" width="1569" height="100" x="0" y="-500"></rect></g></g></g><g transform="translate(1551,0)"><g transform="translate(0,9564)"><g><rect fill="black" stroke="none" width="41597" height="100" x="0" y="500"></rect></g></g><g transform="translate(0,8264)"><use xlink:href="#E269-MJMAIN-31"></use><use xlink:href="#E269-MJMAIN-2E" x="500" y="0"></use><g transform="translate(778,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(1608,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">初</text></g><g transform="translate(2439,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">始</text></g><g transform="translate(3269,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">化</text></g><g transform="translate(4100,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(4931,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">任</text></g><g transform="translate(5761,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">意</text></g><g transform="translate(6592,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">初</text></g><g transform="translate(7423,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">始</text></g><g transform="translate(8253,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">化</text></g><g transform="translate(9084,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">参</text></g><g transform="translate(9915,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">数</text></g><use xlink:href="#E269-MJMAINB-77" x="10995" y="0"></use><g transform="translate(11826,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">。</text></g></g></g><g transform="translate(0,6964)"><use xlink:href="#E269-MJMAIN-32"></use><use xlink:href="#E269-MJMAIN-2E" x="500" y="0"></use><g transform="translate(778,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(1608,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">时</text></g><g transform="translate(2439,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">序</text></g><g transform="translate(3269,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">差</text></g><g transform="translate(4100,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">分</text></g><g transform="translate(4931,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">更</text></g><g transform="translate(5761,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">新</text></g><g transform="translate(6592,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(7423,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">对</text></g><g transform="translate(8253,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">于</text></g><g transform="translate(9084,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">每</text></g><g transform="translate(9915,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">个</text></g><g transform="translate(10745,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">回</text></g><g transform="translate(11576,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">合</text></g><g transform="translate(12407,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">执</text></g><g transform="translate(13237,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">行</text></g><g transform="translate(14068,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">以</text></g><g transform="translate(14899,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">下</text></g><g transform="translate(15729,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">操</text></g><g transform="translate(16560,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">作</text></g><g transform="translate(17391,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">：</text></g></g><g transform="translate(0,5664)"><g transform="translate(2000,0)"><use xlink:href="#E269-MJMAIN-32"></use><use xlink:href="#E269-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E269-MJMAIN-31" x="778" y="0"></use><g transform="translate(1278,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(2108,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">初</text></g><g transform="translate(2939,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">始</text></g><g transform="translate(3769,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">化</text></g><g transform="translate(4600,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">状</text></g><g transform="translate(5431,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">态</text></g><g transform="translate(6261,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(7092,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">选</text></g><g transform="translate(7923,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">择</text></g><g transform="translate(8753,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">状</text></g><g transform="translate(9584,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">态</text></g><use xlink:href="#E269-MJMATHI-53" x="10665" y="0"></use><g transform="translate(11310,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">。</text></g></g></g></g><g transform="translate(0,4364)"><g transform="translate(2000,0)"><use xlink:href="#E269-MJMAIN-32"></use><use xlink:href="#E269-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E269-MJMAIN-32" x="778" y="0"></use><g transform="translate(1972,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">若</text><g transform="translate(830,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">回</text></g><g transform="translate(1661,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">合</text></g><g transform="translate(2491,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">未</text></g><g transform="translate(3322,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">结</text></g><g transform="translate(4153,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">束</text></g><g transform="translate(4983,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">，</text></g><g transform="translate(5814,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">执</text></g><g transform="translate(6645,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">行</text></g><g transform="translate(7475,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">以</text></g><g transform="translate(8306,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">下</text></g><g transform="translate(9137,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">操</text></g><g transform="translate(9967,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">作</text></g><g transform="translate(10798,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">：</text></g></g></g></g><g transform="translate(0,3064)"><g transform="translate(4000,0)"><use xlink:href="#E269-MJMAIN-32"></use><use xlink:href="#E269-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E269-MJMAIN-32" x="778" y="0"></use><use xlink:href="#E269-MJMAIN-2E" x="1278" y="0"></use><use xlink:href="#E269-MJMAIN-31" x="1556" y="0"></use><g transform="translate(2750,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">用</text></g><g transform="translate(3831,0)"><use xlink:href="#E269-MJMATHI-3C0" x="0" y="0"></use><use xlink:href="#E269-MJMAIN-28" x="573" y="0"></use><use xlink:href="#E269-MJMAIN-22C5" x="962" y="0"></use><use xlink:href="#E269-MJMAIN-2223" x="1517" y="0"></use><use xlink:href="#E269-MJMATHI-53" x="2073" y="0"></use><use xlink:href="#E269-MJMAIN-29" x="2718" y="0"></use></g><g transform="translate(6938,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">或</text></g></g><g transform="translate(8269,0)"><use xlink:href="#E269-MJMATHI-71" x="0" y="0"></use><use xlink:href="#E269-MJMAIN-28" x="460" y="0"></use><use xlink:href="#E269-MJMATHI-53" x="849" y="0"></use><use xlink:href="#E269-MJMAIN-2C" x="1494" y="0"></use><use xlink:href="#E269-MJMAIN-22C5" x="1938" y="0"></use><use xlink:href="#E269-MJMAIN-3B" x="2216" y="0"></use><use xlink:href="#E269-MJMAINB-77" x="2661" y="0"></use><use xlink:href="#E269-MJMAIN-29" x="3492" y="0"></use></g><g transform="translate(12150,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">确</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">定</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">动</text></g><g transform="translate(2741,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">作</text></g></g><use xlink:href="#E269-MJMATHI-41" x="15973" y="0"></use><g transform="translate(16723,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">；</text></g></g></g></g><g transform="translate(0,1705)"><g transform="translate(4000,0)"><use xlink:href="#E269-MJMAIN-32"></use><use xlink:href="#E269-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E269-MJMAIN-32" x="778" y="0"></use><use xlink:href="#E269-MJMAIN-2E" x="1278" y="0"></use><use xlink:href="#E269-MJMAIN-32" x="1556" y="0"></use><g transform="translate(2056,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(2886,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">采</text></g><g transform="translate(3717,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">样</text></g><g transform="translate(4547,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(5378,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">执</text></g><g transform="translate(6209,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">行</text></g><g transform="translate(7039,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">动</text></g><g transform="translate(7870,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">作</text></g><use xlink:href="#E269-MJMATHI-41" x="8951" y="0"></use><g transform="translate(9701,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">，</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">观</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">测</text></g><g transform="translate(2741,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">得</text></g><g transform="translate(3572,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">到</text></g><g transform="translate(4403,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">的</text></g><g transform="translate(5233,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">奖</text></g><g transform="translate(6064,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">励</text></g></g><use xlink:href="#E269-MJMATHI-52" x="16846" y="0"></use><g transform="translate(17605,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">和</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">新</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">状</text></g><g transform="translate(2741,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">态</text></g></g><g transform="translate(21428,0)"><use xlink:href="#E269-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E269-MJMAIN-2032" x="925" y="583"></use></g><g transform="translate(22377,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">；</text></g></g></g></g><g transform="translate(0,396)"><g transform="translate(4000,0)"><use xlink:href="#E269-MJMAIN-32"></use><use xlink:href="#E269-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E269-MJMAIN-32" x="778" y="0"></use><use xlink:href="#E269-MJMAIN-2E" x="1278" y="0"></use><use xlink:href="#E269-MJMAIN-33" x="1556" y="0"></use><g transform="translate(2056,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(2886,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">计</text></g><g transform="translate(3717,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">算</text></g><g transform="translate(4547,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">回</text></g><g transform="translate(5378,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">报</text></g><g transform="translate(6209,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">的</text></g><g transform="translate(7039,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">估</text></g><g transform="translate(7870,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">计</text></g><g transform="translate(8701,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">值</text></g><g transform="translate(9531,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(10362,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">状</text></g><g transform="translate(11193,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">态</text></g><g transform="translate(12023,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">价</text></g><g transform="translate(12854,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">值</text></g><g transform="translate(13685,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">评</text></g><g transform="translate(14515,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">估</text></g><g transform="translate(15346,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">：</text></g><g transform="translate(16177,0)"><use xlink:href="#E269-MJMATHI-55" x="0" y="0"></use><use xlink:href="#E269-MJMAIN-2190" x="1044" y="0"></use><use xlink:href="#E269-MJMATHI-52" x="2322" y="0"></use><use xlink:href="#E269-MJMAIN-2B" x="3303" y="0"></use><use xlink:href="#E269-MJMATHI-3B3" x="4304" y="0"></use><use xlink:href="#E269-MJMATHI-76" x="4847" y="0"></use><use xlink:href="#E269-MJMAIN-28" x="5332" y="0"></use><g transform="translate(5721,0)"><use xlink:href="#E269-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E269-MJMAIN-2032" x="925" y="583"></use></g><use xlink:href="#E269-MJMAIN-3B" x="6670" y="0"></use><use xlink:href="#E269-MJMAINB-77" x="7114" y="0"></use><use xlink:href="#E269-MJMAIN-29" x="7945" y="0"></use></g><g transform="translate(24511,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">，</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">期</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">望</text></g><use xlink:href="#E269-MJMAIN-53" x="2991" y="0"></use><use xlink:href="#E269-MJMAIN-41" x="3547" y="0"></use><use xlink:href="#E269-MJMAIN-52" x="4297" y="0"></use><use xlink:href="#E269-MJMAIN-53" x="5033" y="0"></use><use xlink:href="#E269-MJMAIN-41" x="5589" y="0"></use><g transform="translate(6589,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">算</text></g><g transform="translate(7420,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">法</text></g><g transform="translate(8251,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">：</text></g></g></g></g><g transform="translate(0,-1104)"><g transform="translate(6444,0)"><use xlink:href="#E269-MJMATHI-55" x="0" y="0"></use><use xlink:href="#E269-MJMAIN-2190" x="1044" y="0"></use><use xlink:href="#E269-MJMATHI-52" x="2322" y="0"></use><use xlink:href="#E269-MJMAIN-2B" x="3303" y="0"></use><use xlink:href="#E269-MJMATHI-3B3" x="4304" y="0"></use><g transform="translate(5013,0)"><use xlink:href="#E269-MJSZ2-2211" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E269-MJMATHI-61" x="756" y="-1485"></use></g><use xlink:href="#E269-MJMATHI-3C0" x="6624" y="0"></use><use xlink:href="#E269-MJMAIN-28" x="7197" y="0"></use><use xlink:href="#E269-MJMATHI-61" x="7586" y="0"></use><use xlink:href="#E269-MJMAIN-2223" x="8393" y="0"></use><g transform="translate(8948,0)"><use xlink:href="#E269-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E269-MJMAIN-2032" x="925" y="583"></use></g><use xlink:href="#E269-MJMAIN-3B" x="9897" y="0"></use><use xlink:href="#E269-MJMAINB-77" x="10342" y="0"></use><use xlink:href="#E269-MJMAIN-29" x="11173" y="0"></use><use xlink:href="#E269-MJMATHI-71" x="11562" y="0"></use><use xlink:href="#E269-MJMAIN-28" x="12022" y="0"></use><g transform="translate(12411,0)"><use xlink:href="#E269-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E269-MJMAIN-2032" x="925" y="583"></use></g><use xlink:href="#E269-MJMAIN-2C" x="13360" y="0"></use><use xlink:href="#E269-MJMATHI-61" x="13805" y="0"></use><use xlink:href="#E269-MJMAIN-3B" x="14334" y="0"></use><use xlink:href="#E269-MJMAINB-77" x="14779" y="0"></use><use xlink:href="#E269-MJMAIN-29" x="15610" y="0"></use><g transform="translate(15999,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">，</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">其</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">中</text></g></g><g transform="translate(18991,0)"><use xlink:href="#E269-MJMATHI-3C0" x="0" y="0"></use><use xlink:href="#E269-MJMAIN-28" x="573" y="0"></use><use xlink:href="#E269-MJMAIN-22C5" x="962" y="0"></use><use xlink:href="#E269-MJMAIN-2223" x="1517" y="0"></use><g transform="translate(2073,0)"><use xlink:href="#E269-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E269-MJMAIN-2032" x="925" y="583"></use></g><use xlink:href="#E269-MJMAIN-3B" x="3022" y="0"></use><use xlink:href="#E269-MJMAINB-77" x="3467" y="0"></use><use xlink:href="#E269-MJMAIN-29" x="4298" y="0"></use></g><g transform="translate(23678,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">是</text></g></g><g transform="translate(25009,0)"><use xlink:href="#E269-MJMATHI-71" x="0" y="0"></use><use xlink:href="#E269-MJMAIN-28" x="460" y="0"></use><g transform="translate(849,0)"><use xlink:href="#E269-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E269-MJMAIN-2032" x="925" y="583"></use></g><use xlink:href="#E269-MJMAIN-2C" x="1798" y="0"></use><use xlink:href="#E269-MJMAIN-22C5" x="2242" y="0"></use><use xlink:href="#E269-MJMAIN-3B" x="2520" y="0"></use><use xlink:href="#E269-MJMAINB-77" x="2965" y="0"></use><use xlink:href="#E269-MJMAIN-29" x="3796" y="0"></use></g><g transform="translate(29194,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">确</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">定</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">的</text></g></g></g></g><g transform="translate(0,-3370)"><g transform="translate(6444,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">策</text><g transform="translate(830,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">略</text></g><g transform="translate(1661,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(2491,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">如</text></g><use xlink:href="#E269-MJMATHI-3B5" x="3572" y="0"></use><g transform="translate(4038,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">柔</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">性</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">策</text></g><g transform="translate(2741,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">略</text></g><g transform="translate(3572,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(4403,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">，</text></g><use xlink:href="#E269-MJMAIN-51" x="5233" y="0"></use><g transform="translate(6261,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">学</text></g><g transform="translate(7092,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">习</text></g><g transform="translate(7923,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">：</text></g></g><g transform="translate(12792,0)"><use xlink:href="#E269-MJMATHI-55" x="0" y="0"></use><use xlink:href="#E269-MJMAIN-2190" x="1044" y="0"></use><use xlink:href="#E269-MJMATHI-52" x="2322" y="0"></use><use xlink:href="#E269-MJMAIN-2B" x="3303" y="0"></use><use xlink:href="#E269-MJMATHI-3B3" x="4304" y="0"></use><g transform="translate(5013,0)"><use xlink:href="#E269-MJMAIN-6D"></use><use xlink:href="#E269-MJMAIN-61" x="833" y="0"></use><use xlink:href="#E269-MJMAIN-78" x="1333" y="0"></use><use transform="scale(0.707)" xlink:href="#E269-MJMATHI-61" x="1051" y="-865"></use></g><use xlink:href="#E269-MJMATHI-71" x="7208" y="0"></use><use xlink:href="#E269-MJMAIN-28" x="7668" y="0"></use><g transform="translate(8057,0)"><use xlink:href="#E269-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E269-MJMAIN-2032" x="925" y="583"></use></g><use xlink:href="#E269-MJMAIN-2C" x="9006" y="0"></use><use xlink:href="#E269-MJMATHI-61" x="9450" y="0"></use><use xlink:href="#E269-MJMAIN-3B" x="9979" y="0"></use><use xlink:href="#E269-MJMAINB-77" x="10424" y="0"></use></g><g transform="translate(24047,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text><g transform="translate(830,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">；</text></g></g></g></g><g transform="translate(0,-5272)"><g transform="translate(4000,0)"><use xlink:href="#E269-MJMAIN-32"></use><use xlink:href="#E269-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E269-MJMAIN-32" x="778" y="0"></use><use xlink:href="#E269-MJMAIN-2E" x="1278" y="0"></use><use xlink:href="#E269-MJMAIN-34" x="1556" y="0"></use><g transform="translate(2056,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(2886,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">更</text></g><g transform="translate(3717,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">新</text></g><g transform="translate(4547,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">价</text></g><g transform="translate(5378,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">值</text></g><g transform="translate(6209,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(7039,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">状</text></g><g transform="translate(7870,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">态</text></g><g transform="translate(8701,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">价</text></g><g transform="translate(9531,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">值</text></g><g transform="translate(10362,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">评</text></g><g transform="translate(11193,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">估</text></g><g transform="translate(12023,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">：</text></g><g transform="translate(12854,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">更</text></g><g transform="translate(13685,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">新</text></g><use xlink:href="#E269-MJMAINB-77" x="14765" y="0"></use><g transform="translate(15596,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">以</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">减</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">小</text></g></g><g transform="translate(18588,0)"><use xlink:href="#E269-MJMAIN-5B" x="0" y="0"></use><use xlink:href="#E269-MJMATHI-55" x="278" y="0"></use><use xlink:href="#E269-MJMAIN-2212" x="1267" y="0"></use><use xlink:href="#E269-MJMATHI-76" x="2267" y="0"></use><use xlink:href="#E269-MJMAIN-28" x="2752" y="0"></use><use xlink:href="#E269-MJMATHI-53" x="3141" y="0"></use><use xlink:href="#E269-MJMAIN-3B" x="3786" y="0"></use><use xlink:href="#E269-MJMAINB-77" x="4231" y="0"></use><use xlink:href="#E269-MJMAIN-29" x="5062" y="0"></use><g transform="translate(5451,0)"><use xlink:href="#E269-MJMAIN-5D" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E269-MJMAIN-32" x="393" y="583"></use></g></g><g transform="translate(24771,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">，</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">期</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">望</text></g><use xlink:href="#E269-MJMAIN-53" x="2991" y="0"></use><use xlink:href="#E269-MJMAIN-41" x="3547" y="0"></use><use xlink:href="#E269-MJMAIN-52" x="4297" y="0"></use><use xlink:href="#E269-MJMAIN-53" x="5033" y="0"></use><use xlink:href="#E269-MJMAIN-41" x="5589" y="0"></use><g transform="translate(6589,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">算</text></g><g transform="translate(7420,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">法</text></g></g></g></g><g transform="translate(0,-6706)"><g transform="translate(6444,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">和</text><use xlink:href="#E269-MJMAIN-51" x="1080" y="0"></use><g transform="translate(2108,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">学</text></g><g transform="translate(2939,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">习</text></g><g transform="translate(3769,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">：</text></g><g transform="translate(4600,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">更</text></g><g transform="translate(5431,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">新</text></g><use xlink:href="#E269-MJMAINB-77" x="6511" y="0"></use><g transform="translate(7342,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">以</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">减</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">小</text></g></g><g transform="translate(10334,0)"><use xlink:href="#E269-MJMAIN-5B" x="0" y="0"></use><use xlink:href="#E269-MJMATHI-55" x="278" y="0"></use><use xlink:href="#E269-MJMAIN-2212" x="1267" y="0"></use><use xlink:href="#E269-MJMATHI-71" x="2267" y="0"></use><use xlink:href="#E269-MJMAIN-28" x="2727" y="0"></use><use xlink:href="#E269-MJMATHI-53" x="3116" y="0"></use><use xlink:href="#E269-MJMAIN-2C" x="3761" y="0"></use><use xlink:href="#E269-MJMATHI-41" x="4206" y="0"></use><use xlink:href="#E269-MJMAIN-3B" x="4956" y="0"></use><use xlink:href="#E269-MJMAINB-77" x="5400" y="0"></use><use xlink:href="#E269-MJMAIN-29" x="6231" y="0"></use><g transform="translate(6620,0)"><use xlink:href="#E269-MJMAIN-5D" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E269-MJMAIN-32" x="393" y="583"></use></g></g><g transform="translate(17687,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">；</text></g></g></g></g><g transform="translate(0,-8065)"><g transform="translate(4000,0)"><use xlink:href="#E269-MJMAIN-32"></use><use xlink:href="#E269-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E269-MJMAIN-32" x="778" y="0"></use><use xlink:href="#E269-MJMAIN-2E" x="1278" y="0"></use><use xlink:href="#E269-MJMAIN-35" x="1556" y="0"></use><g transform="translate(2306,0)"><use xlink:href="#E269-MJMATHI-53" x="444" y="0"></use><use xlink:href="#E269-MJMAIN-2190" x="1367" y="0"></use><g transform="translate(2645,0)"><use xlink:href="#E269-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E269-MJMAIN-2032" x="925" y="583"></use></g></g><g transform="translate(5900,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">。</text></g></g></g></g><g transform="translate(0,-9365)"><g><rect fill="black" stroke="none" width="41597" height="100" x="0" y="-500"></rect></g></g></g></g></g></svg></span></div><script type="math/tex; mode=display" id="MathJax-Element-242">\; \\ \; \\
\large \textbf{算法 6-4   半梯度下降算法估计状态价值或期望 SARSA 算法或 Q 学习} \\
\begin{split}
\rule[5pt]{10mm}{0.1em} &\rule[5pt]{265mm}{0.1em} \\
&\text{1.（初始化）任意初始化参数 $\bold w$ 。} \\
&\text{2.（时序差分更新）对于每个回合执行以下操作：} \\
&\qquad \text{2.1（初始化状态）选择状态 $S$ 。} \\
&\qquad \text{2.2 $\;\,$若回合未结束，执行以下操作：} \\
&\qquad \qquad \text{2.2.1 $\;\,$用 $\pi(\cdot \mid S)$ 或 $q(S,\cdot;\bold w)$ 确定动作 $A$ ；} \\
&\qquad \qquad \text{2.2.2（采样）执行动作 $A$ ，观测得到的奖励 $R$ 和新状态 $S'$ ；} \\
&\qquad \qquad \text{2.2.3（计算回报的估计值）状态价值评估：$U \leftarrow R + \gamma v(S';\bold w)$ ，期望 SARSA 算法：} \\
&\qquad \qquad \qquad \;\, \text{$U \leftarrow R + \gamma \sum_a \pi(a \mid S';\bold w) q(S',a;\bold w)$ ，其中 $\pi(\cdot \mid S';\bold w)$ 是 $q(S',\cdot;\bold w)$ 确定的} \\
&\qquad \qquad \qquad \;\, \text{策略（如 $\varepsilon$ 柔性策略），Q 学习：$U \leftarrow R + \gamma \max_a\, q(S',a;\bold w$）；} \\
&\qquad \qquad \text{2.2.4（更新价值）状态价值评估：更新 $\bold w$ 以减小 $[U-v(S;\bold w)]^2$ ，期望 SARSA 算法} \\
&\qquad \qquad \qquad \;\, \text{和 Q 学习：更新 $\bold w$ 以减小 $[U-q(S,A;\bold w)]^2$ ；}\\
&\qquad \qquad \text{2.2.5 $\;\, S \leftarrow S'$ 。}\\
\rule[-5pt]{10mm}{0.1em} &\rule[-5pt]{265mm}{0.1em}
\end{split}
\; \\ \; \\</script></div></div><p><span>需要注意的是，当采用自动计算微分并更新参数的软件包来减小损失时，则务必注意不能对回报的估计求梯度。</span></p><p><span>资格迹同样可以运用在函数近似算法中，实现回合更新和单步时序差分的折中。这时的资格迹参数 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="1.187ex" height="1.263ex" viewBox="0 -500.4 511 543.9" role="img" focusable="false" style="vertical-align: -0.101ex;"><defs><path stroke-width="0" id="E308-MJMAINB-7A" d="M48 262Q48 264 54 349T60 436V444H252Q289 444 336 444T394 445Q441 445 450 441T459 418Q459 406 458 404Q456 399 327 229T194 55H237Q260 56 268 56T297 58T325 65T348 77T370 98T384 128T395 170Q400 197 400 216Q400 217 431 217H462V211Q461 208 453 108T444 6V0H245Q46 0 43 2Q32 7 32 28V33Q32 41 40 52T84 112Q129 170 164 217L298 393H256Q189 392 165 380Q124 360 115 303Q110 280 110 256Q110 254 79 254H48V262Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E308-MJMAINB-7A" x="0" y="0"></use></g></svg></span><script type="math/tex">\bold z</script><span> 和价值参数 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="1.93ex" height="1.36ex" viewBox="0 -500.4 831 585.5" role="img" focusable="false" style="vertical-align: -0.198ex;"><defs><path stroke-width="0" id="E337-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E337-MJMAINB-77" x="0" y="0"></use></g></svg></span><script type="math/tex">\bold w</script><span> 具有相同形状的大小，并且逐元素一一对应；也就是说资格迹参数表示了在更新价值参数时应当使用的权重乘以价值估计的梯度，那么价值参数的更新式应当如下：</span></p><div contenteditable="false" spellcheck="false" class="mathjax-block md-end-block md-math-block md-rawblock" id="mathjax-n19" cid="n19" mdtype="math_block"><div class="md-rawblock-container md-math-container" tabindex="-1"><div class="MathJax_SVG_Display" style="text-align: center;"><span class="MathJax_SVG" id="MathJax-Element-243-Frame" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="49.095ex" height="5.797ex" viewBox="0 -1497.2 21137.9 2496" role="img" focusable="false" style="vertical-align: -2.32ex; max-width: 100%;"><defs><path stroke-width="0" id="E270-MJMAIN-7B" d="M434 -231Q434 -244 428 -250H410Q281 -250 230 -184Q225 -177 222 -172T217 -161T213 -148T211 -133T210 -111T209 -84T209 -47T209 0Q209 21 209 53Q208 142 204 153Q203 154 203 155Q189 191 153 211T82 231Q71 231 68 234T65 250T68 266T82 269Q116 269 152 289T203 345Q208 356 208 377T209 529V579Q209 634 215 656T244 698Q270 724 324 740Q361 748 377 749Q379 749 390 749T408 750H428Q434 744 434 732Q434 719 431 716Q429 713 415 713Q362 710 332 689T296 647Q291 634 291 499V417Q291 370 288 353T271 314Q240 271 184 255L170 250L184 245Q202 239 220 230T262 196T290 137Q291 131 291 1Q291 -134 296 -147Q306 -174 339 -192T415 -213Q429 -213 431 -216Q434 -219 434 -231Z"></path><path stroke-width="0" id="E270-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E270-MJMAIN-2190" d="M944 261T944 250T929 230H165Q167 228 182 216T211 189T244 152T277 96T303 25Q308 7 308 0Q308 -11 288 -11Q281 -11 278 -11T272 -7T267 2T263 21Q245 94 195 151T73 236Q58 242 55 247Q55 254 59 257T73 264Q121 283 158 314T215 375T247 434T264 480L267 497Q269 503 270 505T275 509T288 511Q308 511 308 500Q308 493 303 475Q293 438 278 406T246 352T215 315T185 287T165 270H929Q944 261 944 250Z"></path><path stroke-width="0" id="E270-MJMAIN-2B" d="M56 237T56 250T70 270H369V420L370 570Q380 583 389 583Q402 583 409 568V270H707Q722 262 722 250T707 230H409V-68Q401 -82 391 -82H389H387Q375 -82 369 -68V230H70Q56 237 56 250Z"></path><path stroke-width="0" id="E270-MJMATHI-3B1" d="M34 156Q34 270 120 356T309 442Q379 442 421 402T478 304Q484 275 485 237V208Q534 282 560 374Q564 388 566 390T582 393Q603 393 603 385Q603 376 594 346T558 261T497 161L486 147L487 123Q489 67 495 47T514 26Q528 28 540 37T557 60Q559 67 562 68T577 70Q597 70 597 62Q597 56 591 43Q579 19 556 5T512 -10H505Q438 -10 414 62L411 69L400 61Q390 53 370 41T325 18T267 -2T203 -11Q124 -11 79 39T34 156ZM208 26Q257 26 306 47T379 90L403 112Q401 255 396 290Q382 405 304 405Q235 405 183 332Q156 292 139 224T121 120Q121 71 146 49T208 26Z"></path><path stroke-width="0" id="E270-MJMAIN-5B" d="M118 -250V750H255V710H158V-210H255V-250H118Z"></path><path stroke-width="0" id="E270-MJMATHI-55" d="M107 637Q73 637 71 641Q70 643 70 649Q70 673 81 682Q83 683 98 683Q139 681 234 681Q268 681 297 681T342 682T362 682Q378 682 378 672Q378 670 376 658Q371 641 366 638H364Q362 638 359 638T352 638T343 637T334 637Q295 636 284 634T266 623Q265 621 238 518T184 302T154 169Q152 155 152 140Q152 86 183 55T269 24Q336 24 403 69T501 205L552 406Q599 598 599 606Q599 633 535 637Q511 637 511 648Q511 650 513 660Q517 676 519 679T529 683Q532 683 561 682T645 680Q696 680 723 681T752 682Q767 682 767 672Q767 650 759 642Q756 637 737 637Q666 633 648 597Q646 592 598 404Q557 235 548 205Q515 105 433 42T263 -22Q171 -22 116 34T60 167V183Q60 201 115 421Q164 622 164 628Q164 635 107 637Z"></path><path stroke-width="0" id="E270-MJMAIN-2212" d="M84 237T84 250T98 270H679Q694 262 694 250T679 230H98Q84 237 84 250Z"></path><path stroke-width="0" id="E270-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E270-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E270-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E270-MJMATHI-74" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path><path stroke-width="0" id="E270-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E270-MJMATHI-41" d="M208 74Q208 50 254 46Q272 46 272 35Q272 34 270 22Q267 8 264 4T251 0Q249 0 239 0T205 1T141 2Q70 2 50 0H42Q35 7 35 11Q37 38 48 46H62Q132 49 164 96Q170 102 345 401T523 704Q530 716 547 716H555H572Q578 707 578 706L606 383Q634 60 636 57Q641 46 701 46Q726 46 726 36Q726 34 723 22Q720 7 718 4T704 0Q701 0 690 0T651 1T578 2Q484 2 455 0H443Q437 6 437 9T439 27Q443 40 445 43L449 46H469Q523 49 533 63L521 213H283L249 155Q208 86 208 74ZM516 260Q516 271 504 416T490 562L463 519Q447 492 400 412L310 260L413 259Q516 259 516 260Z"></path><path stroke-width="0" id="E270-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E270-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E270-MJMAIN-5D" d="M22 710V750H159V-250H22V-210H119V710H22Z"></path><path stroke-width="0" id="E270-MJMAINB-7A" d="M48 262Q48 264 54 349T60 436V444H252Q289 444 336 444T394 445Q441 445 450 441T459 418Q459 406 458 404Q456 399 327 229T194 55H237Q260 56 268 56T297 58T325 65T348 77T370 98T384 128T395 170Q400 197 400 216Q400 217 431 217H462V211Q461 208 453 108T444 6V0H245Q46 0 43 2Q32 7 32 28V33Q32 41 40 52T84 112Q129 170 164 217L298 393H256Q189 392 165 380Q124 360 115 303Q110 280 110 256Q110 254 79 254H48V262Z"></path><path stroke-width="0" id="E270-MJMATHI-76" d="M173 380Q173 405 154 405Q130 405 104 376T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Q21 294 29 316T53 368T97 419T160 441Q202 441 225 417T249 361Q249 344 246 335Q246 329 231 291T200 202T182 113Q182 86 187 69Q200 26 250 26Q287 26 319 60T369 139T398 222T409 277Q409 300 401 317T383 343T365 361T357 383Q357 405 376 424T417 443Q436 443 451 425T467 367Q467 340 455 284T418 159T347 40T241 -11Q177 -11 139 22Q102 54 102 117Q102 148 110 181T151 298Q173 362 173 380Z"></path><path stroke-width="0" id="E270-MJSZ3-7B" d="M618 -943L612 -949H582L568 -943Q472 -903 411 -841T332 -703Q327 -682 327 -653T325 -350Q324 -28 323 -18Q317 24 301 61T264 124T221 171T179 205T147 225T132 234Q130 238 130 250Q130 255 130 258T131 264T132 267T134 269T139 272T144 275Q207 308 256 367Q310 436 323 519Q324 529 325 851Q326 1124 326 1154T332 1205Q369 1358 566 1443L582 1450H612L618 1444V1429Q618 1413 616 1411L608 1406Q599 1402 585 1393T552 1372T515 1343T479 1305T449 1257T429 1200Q425 1180 425 1152T423 851Q422 579 422 549T416 498Q407 459 388 424T346 364T297 318T250 284T214 264T197 254L188 251L205 242Q290 200 345 138T416 3Q421 -18 421 -48T423 -349Q423 -397 423 -472Q424 -677 428 -694Q429 -697 429 -699Q434 -722 443 -743T465 -782T491 -816T519 -845T548 -868T574 -886T595 -899T610 -908L616 -910Q618 -912 618 -928V-943Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E270-MJSZ3-7B"></use><g transform="translate(917,0)"><g transform="translate(-19,0)"><g transform="translate(30,650)"><use xlink:href="#E270-MJMAINB-77" x="0" y="0"></use><use xlink:href="#E270-MJMAIN-2190" x="1108" y="0"></use><use xlink:href="#E270-MJMAINB-77" x="2386" y="0"></use><use xlink:href="#E270-MJMAIN-2B" x="3439" y="0"></use><use xlink:href="#E270-MJMATHI-3B1" x="4440" y="0"></use><use xlink:href="#E270-MJMAIN-5B" x="5080" y="0"></use><use xlink:href="#E270-MJMATHI-55" x="5358" y="0"></use><use xlink:href="#E270-MJMAIN-2212" x="6347" y="0"></use><use xlink:href="#E270-MJMATHI-71" x="7347" y="0"></use><use xlink:href="#E270-MJMAIN-28" x="7807" y="0"></use><g transform="translate(8196,0)"><use xlink:href="#E270-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E270-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E270-MJMAIN-2C" x="9164" y="0"></use><g transform="translate(9609,0)"><use xlink:href="#E270-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E270-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E270-MJMAIN-3B" x="10714" y="0"></use><use xlink:href="#E270-MJMAINB-77" x="11159" y="0"></use><use xlink:href="#E270-MJMAIN-29" x="11990" y="0"></use><use xlink:href="#E270-MJMAIN-5D" x="12379" y="0"></use><use xlink:href="#E270-MJMAINB-7A" x="12657" y="0"></use><use xlink:href="#E270-MJMAIN-2C" x="13612" y="0"></use><g transform="translate(15057,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">更</text><g transform="translate(830,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">新</text></g><g transform="translate(1661,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">动</text></g><g transform="translate(2491,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">作</text></g><g transform="translate(3322,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">价</text></g><g transform="translate(4153,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">值</text></g></g></g><g transform="translate(0,-700)"><use xlink:href="#E270-MJMAINB-77" x="0" y="0"></use><use xlink:href="#E270-MJMAIN-2190" x="1108" y="0"></use><use xlink:href="#E270-MJMAINB-77" x="2386" y="0"></use><use xlink:href="#E270-MJMAIN-2B" x="3439" y="0"></use><use xlink:href="#E270-MJMATHI-3B1" x="4440" y="0"></use><use xlink:href="#E270-MJMAIN-5B" x="5080" y="0"></use><use xlink:href="#E270-MJMATHI-55" x="5358" y="0"></use><use xlink:href="#E270-MJMAIN-2212" x="6347" y="0"></use><use xlink:href="#E270-MJMATHI-76" x="7347" y="0"></use><use xlink:href="#E270-MJMAIN-28" x="7832" y="0"></use><g transform="translate(8221,0)"><use xlink:href="#E270-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E270-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E270-MJMAIN-3B" x="9189" y="0"></use><use xlink:href="#E270-MJMAINB-77" x="9634" y="0"></use><use xlink:href="#E270-MJMAIN-29" x="10465" y="0"></use><use xlink:href="#E270-MJMAIN-5D" x="10854" y="0"></use><use xlink:href="#E270-MJMAINB-7A" x="11132" y="0"></use><use xlink:href="#E270-MJMAIN-2C" x="12087" y="0"></use><g transform="translate(15088,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">更</text><g transform="translate(830,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">新</text></g><g transform="translate(1661,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">动</text></g><g transform="translate(2491,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">作</text></g><g transform="translate(3322,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">价</text></g><g transform="translate(4153,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">值</text></g></g></g></g></g></g></svg></span></div><script type="math/tex; mode=display" id="MathJax-Element-243">\left \{
\begin{aligned}
\bold w \leftarrow \bold w + \alpha[U - q(S_t,A_t;\bold w)] \bold z \;\, , \quad \text{更新动作价值}\\
\bold w \leftarrow \bold w + \alpha[U - v(S_t;\bold w)] \bold z \;\, , \;\;\qquad \text{更新动作价值}\\
\end{aligned}
\right.</script></div></div><p><span>当资格迹为累积迹时，其定义如下：</span></p><div contenteditable="false" spellcheck="false" class="mathjax-block md-end-block md-math-block md-rawblock" id="mathjax-n21" cid="n21" mdtype="math_block"><div class="md-rawblock-container md-math-container" tabindex="-1"><div class="MathJax_SVG_Display" style="text-align: center;"><span class="MathJax_SVG" id="MathJax-Element-244-Frame" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="51.722ex" height="8.884ex" viewBox="0 -2161.7 22268.9 3825" role="img" focusable="false" style="vertical-align: -3.863ex; max-width: 100%;"><defs><path stroke-width="0" id="E271-MJMAIN-7B" d="M434 -231Q434 -244 428 -250H410Q281 -250 230 -184Q225 -177 222 -172T217 -161T213 -148T211 -133T210 -111T209 -84T209 -47T209 0Q209 21 209 53Q208 142 204 153Q203 154 203 155Q189 191 153 211T82 231Q71 231 68 234T65 250T68 266T82 269Q116 269 152 289T203 345Q208 356 208 377T209 529V579Q209 634 215 656T244 698Q270 724 324 740Q361 748 377 749Q379 749 390 749T408 750H428Q434 744 434 732Q434 719 431 716Q429 713 415 713Q362 710 332 689T296 647Q291 634 291 499V417Q291 370 288 353T271 314Q240 271 184 255L170 250L184 245Q202 239 220 230T262 196T290 137Q291 131 291 1Q291 -134 296 -147Q306 -174 339 -192T415 -213Q429 -213 431 -216Q434 -219 434 -231Z"></path><path stroke-width="0" id="E271-MJMAINB-7A" d="M48 262Q48 264 54 349T60 436V444H252Q289 444 336 444T394 445Q441 445 450 441T459 418Q459 406 458 404Q456 399 327 229T194 55H237Q260 56 268 56T297 58T325 65T348 77T370 98T384 128T395 170Q400 197 400 216Q400 217 431 217H462V211Q461 208 453 108T444 6V0H245Q46 0 43 2Q32 7 32 28V33Q32 41 40 52T84 112Q129 170 164 217L298 393H256Q189 392 165 380Q124 360 115 303Q110 280 110 256Q110 254 79 254H48V262Z"></path><path stroke-width="0" id="E271-MJMAIN-30" d="M96 585Q152 666 249 666Q297 666 345 640T423 548Q460 465 460 320Q460 165 417 83Q397 41 362 16T301 -15T250 -22Q224 -22 198 -16T137 16T82 83Q39 165 39 320Q39 494 96 585ZM321 597Q291 629 250 629Q208 629 178 597Q153 571 145 525T137 333Q137 175 145 125T181 46Q209 16 250 16Q290 16 318 46Q347 76 354 130T362 333Q362 478 354 524T321 597Z"></path><path stroke-width="0" id="E271-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E271-MJMAINB-30" d="M266 654H280H282Q500 654 524 418Q529 370 529 320Q529 125 456 52Q397 -10 287 -10Q110 -10 63 154Q45 212 45 316Q45 504 113 585Q140 618 185 636T266 654ZM374 548Q347 604 286 604Q247 604 218 575Q197 552 193 511T188 311Q188 159 196 116Q202 87 225 64T287 41Q339 41 367 87Q379 107 382 152T386 329Q386 518 374 548Z"></path><path stroke-width="0" id="E271-MJMATHI-74" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path><path stroke-width="0" id="E271-MJMATHI-3B3" d="M31 249Q11 249 11 258Q11 275 26 304T66 365T129 418T206 441Q233 441 239 440Q287 429 318 386T371 255Q385 195 385 170Q385 166 386 166L398 193Q418 244 443 300T486 391T508 430Q510 431 524 431H537Q543 425 543 422Q543 418 522 378T463 251T391 71Q385 55 378 6T357 -100Q341 -165 330 -190T303 -216Q286 -216 286 -188Q286 -138 340 32L346 51L347 69Q348 79 348 100Q348 257 291 317Q251 355 196 355Q148 355 108 329T51 260Q49 251 47 251Q45 249 31 249Z"></path><path stroke-width="0" id="E271-MJMATHI-3BB" d="M166 673Q166 685 183 694H202Q292 691 316 644Q322 629 373 486T474 207T524 67Q531 47 537 34T546 15T551 6T555 2T556 -2T550 -11H482Q457 3 450 18T399 152L354 277L340 262Q327 246 293 207T236 141Q211 112 174 69Q123 9 111 -1T83 -12Q47 -12 47 20Q47 37 61 52T199 187Q229 216 266 252T321 306L338 322Q338 323 288 462T234 612Q214 657 183 657Q166 657 166 673Z"></path><path stroke-width="0" id="E271-MJMAIN-2212" d="M84 237T84 250T98 270H679Q694 262 694 250T679 230H98Q84 237 84 250Z"></path><path stroke-width="0" id="E271-MJMAIN-31" d="M213 578L200 573Q186 568 160 563T102 556H83V602H102Q149 604 189 617T245 641T273 663Q275 666 285 666Q294 666 302 660V361L303 61Q310 54 315 52T339 48T401 46H427V0H416Q395 3 257 3Q121 3 100 0H88V46H114Q136 46 152 46T177 47T193 50T201 52T207 57T213 61V578Z"></path><path stroke-width="0" id="E271-MJMAIN-2B" d="M56 237T56 250T70 270H369V420L370 570Q380 583 389 583Q402 583 409 568V270H707Q722 262 722 250T707 230H409V-68Q401 -82 391 -82H389H387Q375 -82 369 -68V230H70Q56 237 56 250Z"></path><path stroke-width="0" id="E271-MJMAIN-2207" d="M46 676Q46 679 51 683H781Q786 679 786 676Q786 674 617 326T444 -26Q439 -33 416 -33T388 -26Q385 -22 216 326T46 676ZM697 596Q697 597 445 597T193 596Q195 591 319 336T445 80L697 596Z"></path><path stroke-width="0" id="E271-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E271-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E271-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E271-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E271-MJMATHI-41" d="M208 74Q208 50 254 46Q272 46 272 35Q272 34 270 22Q267 8 264 4T251 0Q249 0 239 0T205 1T141 2Q70 2 50 0H42Q35 7 35 11Q37 38 48 46H62Q132 49 164 96Q170 102 345 401T523 704Q530 716 547 716H555H572Q578 707 578 706L606 383Q634 60 636 57Q641 46 701 46Q726 46 726 36Q726 34 723 22Q720 7 718 4T704 0Q701 0 690 0T651 1T578 2Q484 2 455 0H443Q437 6 437 9T439 27Q443 40 445 43L449 46H469Q523 49 533 63L521 213H283L249 155Q208 86 208 74ZM516 260Q516 271 504 416T490 562L463 519Q447 492 400 412L310 260L413 259Q516 259 516 260Z"></path><path stroke-width="0" id="E271-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E271-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E271-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E271-MJMATHI-76" d="M173 380Q173 405 154 405Q130 405 104 376T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Q21 294 29 316T53 368T97 419T160 441Q202 441 225 417T249 361Q249 344 246 335Q246 329 231 291T200 202T182 113Q182 86 187 69Q200 26 250 26Q287 26 319 60T369 139T398 222T409 277Q409 300 401 317T383 343T365 361T357 383Q357 405 376 424T417 443Q436 443 451 425T467 367Q467 340 455 284T418 159T347 40T241 -11Q177 -11 139 22Q102 54 102 117Q102 148 110 181T151 298Q173 362 173 380Z"></path><path stroke-width="0" id="E271-MJSZ4-23A7" d="M712 899L718 893V876V865Q718 854 704 846Q627 793 577 710T510 525Q510 524 509 521Q505 493 504 349Q504 345 504 334Q504 277 504 240Q504 -2 503 -4Q502 -8 494 -9T444 -10Q392 -10 390 -9Q387 -8 386 -5Q384 5 384 230Q384 262 384 312T383 382Q383 481 392 535T434 656Q510 806 664 892L677 899H712Z"></path><path stroke-width="0" id="E271-MJSZ4-23A9" d="M718 -893L712 -899H677L666 -893Q542 -825 468 -714T385 -476Q384 -466 384 -282Q384 3 385 5L389 9Q392 10 444 10Q486 10 494 9T503 4Q504 2 504 -239V-310V-366Q504 -470 508 -513T530 -609Q546 -657 569 -698T617 -767T661 -812T699 -843T717 -856T718 -876V-893Z"></path><path stroke-width="0" id="E271-MJSZ4-23A8" d="M389 1159Q391 1160 455 1160Q496 1160 498 1159Q501 1158 502 1155Q504 1145 504 924Q504 691 503 682Q494 549 425 439T243 259L229 250L243 241Q349 175 421 66T503 -182Q504 -191 504 -424Q504 -600 504 -629T499 -659H498Q496 -660 444 -660T390 -659Q387 -658 386 -655Q384 -645 384 -425V-282Q384 -176 377 -116T342 10Q325 54 301 92T255 155T214 196T183 222T171 232Q170 233 170 250T171 268Q171 269 191 284T240 331T300 407T354 524T383 679Q384 691 384 925Q384 1152 385 1155L389 1159Z"></path><path stroke-width="0" id="E271-MJSZ4-23AA" d="M384 150V266Q384 304 389 309Q391 310 455 310Q496 310 498 309Q502 308 503 298Q504 283 504 150Q504 32 504 12T499 -9H498Q496 -10 444 -10T390 -9Q386 -8 385 2Q384 17 384 150Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><g transform="translate(0,2100)"><use xlink:href="#E271-MJSZ4-23A7" x="0" y="-899"></use><g transform="translate(0,-985.90625) scale(1,0.409375)"><use xlink:href="#E271-MJSZ4-23AA"></use></g><use xlink:href="#E271-MJSZ4-23A8" x="0" y="-2100"></use><g transform="translate(0,-2836.90625) scale(1,0.409375)"><use xlink:href="#E271-MJSZ4-23AA"></use></g><use xlink:href="#E271-MJSZ4-23A9" x="0" y="-2801"></use></g><g transform="translate(1056,0)"><g transform="translate(-19,0)"><g transform="translate(167,0)"><g transform="translate(-19,0)"><g transform="translate(0,1300)"><use xlink:href="#E271-MJMAINB-7A" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E271-MJMAIN-30" x="722" y="-213"></use><use xlink:href="#E271-MJMAIN-3D" x="1242" y="0"></use><use xlink:href="#E271-MJMAINB-30" x="2298" y="0"></use></g><use xlink:href="#E271-MJMAINB-7A" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E271-MJMATHI-74" x="722" y="-213"></use><use xlink:href="#E271-MJMAIN-3D" x="1144" y="0"></use><use xlink:href="#E271-MJMATHI-3B3" x="2199" y="0"></use><use xlink:href="#E271-MJMATHI-3BB" x="2742" y="0"></use><g transform="translate(3325,0)"><use xlink:href="#E271-MJMAINB-7A" x="0" y="0"></use><g transform="translate(511,-150)"><use transform="scale(0.707)" xlink:href="#E271-MJMATHI-74" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E271-MJMAIN-2212" x="361" y="0"></use><use transform="scale(0.707)" xlink:href="#E271-MJMAIN-31" x="1139" y="0"></use></g></g><use xlink:href="#E271-MJMAIN-2B" x="5317" y="0"></use><use xlink:href="#E271-MJMAIN-2207" x="6318" y="0"></use><use xlink:href="#E271-MJMATHI-71" x="7151" y="0"></use><use xlink:href="#E271-MJMAIN-28" x="7611" y="0"></use><g transform="translate(8000,0)"><use xlink:href="#E271-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E271-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E271-MJMAIN-2C" x="8968" y="0"></use><g transform="translate(9413,0)"><use xlink:href="#E271-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E271-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E271-MJMAIN-3B" x="10518" y="0"></use><use xlink:href="#E271-MJMAINB-77" x="10963" y="0"></use><use xlink:href="#E271-MJMAIN-29" x="11794" y="0"></use><use xlink:href="#E271-MJMAIN-2C" x="12627" y="0"></use><g transform="translate(14072,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">动</text><g transform="translate(830,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">作</text></g><g transform="translate(1661,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">价</text></g><g transform="translate(2491,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">值</text></g><g transform="translate(3322,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">的</text></g><g transform="translate(4153,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">资</text></g><g transform="translate(4983,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">格</text></g><g transform="translate(5814,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">迹</text></g></g><g transform="translate(0,-1350)"><use xlink:href="#E271-MJMAINB-7A" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E271-MJMATHI-74" x="722" y="-213"></use><use xlink:href="#E271-MJMAIN-3D" x="1144" y="0"></use><use xlink:href="#E271-MJMATHI-3B3" x="2199" y="0"></use><use xlink:href="#E271-MJMATHI-3BB" x="2742" y="0"></use><g transform="translate(3325,0)"><use xlink:href="#E271-MJMAINB-7A" x="0" y="0"></use><g transform="translate(511,-150)"><use transform="scale(0.707)" xlink:href="#E271-MJMATHI-74" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E271-MJMAIN-2212" x="361" y="0"></use><use transform="scale(0.707)" xlink:href="#E271-MJMAIN-31" x="1139" y="0"></use></g></g><use xlink:href="#E271-MJMAIN-2B" x="5317" y="0"></use><use xlink:href="#E271-MJMAIN-2207" x="6318" y="0"></use><use xlink:href="#E271-MJMATHI-76" x="7151" y="0"></use><use xlink:href="#E271-MJMAIN-28" x="7636" y="0"></use><g transform="translate(8025,0)"><use xlink:href="#E271-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E271-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E271-MJMAIN-3B" x="8993" y="0"></use><use xlink:href="#E271-MJMAINB-77" x="9438" y="0"></use><use xlink:href="#E271-MJMAIN-29" x="10269" y="0"></use><use xlink:href="#E271-MJMAIN-2C" x="11102" y="0"></use><g transform="translate(14102,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">状</text><g transform="translate(830,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">态</text></g><g transform="translate(1661,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">价</text></g><g transform="translate(2491,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">值</text></g><g transform="translate(3322,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">的</text></g><g transform="translate(4153,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">资</text></g><g transform="translate(4983,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">格</text></g><g transform="translate(5814,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">迹</text></g></g></g></g></g></g></g></g></svg></span></div><script type="math/tex; mode=display" id="MathJax-Element-244">\left \{
\begin{aligned}
\begin{split}
&\bold z_0 = \bold 0 \\
&\bold z_t = \gamma\lambda\bold z_{t-1} + \nabla q(S_t,A_t;\bold w)\;\, , \quad \text{动作价值的资格迹} \\
&\bold z_t = \gamma\lambda\bold z_{t-1} + \nabla v(S_t;\bold w)\;\, , \;\;\qquad \text{状态价值的资格迹} \\
\end{split}
\end{aligned}
\right.</script></div></div><p><span>根据以上结果，即可得到结合资格迹的函数近似算法：</span></p><div contenteditable="false" spellcheck="false" class="mathjax-block md-end-block md-math-block md-rawblock" id="mathjax-n23" cid="n23" mdtype="math_block"><div class="md-rawblock-container md-math-container" tabindex="-1"><div class="MathJax_SVG_Display"><span class="MathJax_SVG" id="MathJax-Element-245-Frame" tabindex="-1" style="font-size: 100%; display: inline-block; zoom: 0.971345;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="101.294ex" height="30.009ex" viewBox="-18.1 -43.5 43612.5 12920.6" role="img" focusable="false" style="vertical-align: -29.908ex; margin-left: -0.042ex; max-width: 100%;"><defs><path stroke-width="0" id="E272-MJMAINB-36" d="M48 318Q48 395 68 456T120 553T193 613T273 646T350 655Q425 655 461 616T497 524Q497 485 475 468T428 451Q399 451 378 470T357 521Q357 565 403 588Q375 601 351 601Q313 601 282 584Q242 565 222 526Q199 473 199 367Q201 369 210 380T227 396T246 410T275 422T312 426Q438 426 494 332Q526 285 526 208V199Q526 112 465 53Q428 17 388 3T285 -11Q236 -11 195 7T135 43T104 80Q48 165 48 318ZM375 231V244V268Q375 295 373 310T364 342T341 366T299 374H297Q231 374 208 287Q200 257 200 196Q201 120 209 100Q231 47 288 47Q351 47 368 90Q375 112 375 231Z"></path><path stroke-width="0" id="E272-MJMAINB-2D" d="M13 166V278H318V166H13Z"></path><path stroke-width="0" id="E272-MJMAINB-35" d="M100 565V605Q100 637 102 646T113 655Q116 655 139 647T202 631T286 623Q332 623 372 631T434 647T459 655Q466 655 469 651T472 643T472 629Q472 613 463 601Q370 487 219 487Q195 487 183 488T169 490T168 433V376Q169 376 174 379T188 387T211 397T244 405T288 409Q390 409 453 352T517 201Q517 106 445 48T253 -11Q169 -11 113 37T57 154Q57 187 79 208T131 229T183 209T206 154Q206 99 155 83Q152 82 157 78Q196 47 253 47Q347 47 358 135Q358 137 358 138Q360 158 360 209Q360 277 355 301T337 338Q315 358 282 358Q202 358 160 303Q153 294 149 292T130 290Q107 290 102 301Q100 304 100 474V565Z"></path><path stroke-width="0" id="E272-MJMAINB-54" d="M41 425Q41 426 51 545T62 669V675H737V669Q738 665 748 546T758 425V419H696V425Q687 517 669 555T595 607Q578 612 522 613H478V62H631V0H615Q585 3 399 3Q214 3 184 0H168V62H321V613H277H263Q164 613 134 561Q113 527 103 425V419H41V425Z"></path><path stroke-width="0" id="E272-MJMAINB-44" d="M39 624V686H270H310H408Q500 686 545 680T638 649Q768 584 805 438Q817 388 817 338Q817 171 702 75Q628 17 515 2Q504 1 270 0H39V62H147V624H39ZM655 337Q655 370 655 390T650 442T639 494T616 540T580 580T526 607T451 623Q443 624 368 624H298V62H377H387H407Q445 62 472 65T540 83T606 129Q629 156 640 195T653 262T655 337Z"></path><path stroke-width="0" id="E272-MJMAINB-28" d="M103 166T103 251T121 412T165 541T225 639T287 708T341 750H356H361Q382 750 382 736Q382 732 365 714T323 661T274 576T232 439T214 250Q214 -62 381 -229Q382 -231 382 -234Q382 -249 360 -249H356H341Q314 -231 287 -207T226 -138T165 -41T121 89Z"></path><path stroke-width="0" id="E272-MJMATHI-3BB" d="M166 673Q166 685 183 694H202Q292 691 316 644Q322 629 373 486T474 207T524 67Q531 47 537 34T546 15T551 6T555 2T556 -2T550 -11H482Q457 3 450 18T399 152L354 277L340 262Q327 246 293 207T236 141Q211 112 174 69Q123 9 111 -1T83 -12Q47 -12 47 20Q47 37 61 52T199 187Q229 216 266 252T321 306L338 322Q338 323 288 462T234 612Q214 657 183 657Q166 657 166 673Z"></path><path stroke-width="0" id="E272-MJMAINB-29" d="M231 251Q231 354 214 439T173 575T123 661T81 714T64 735Q64 744 73 749H75Q77 749 79 749T84 750T90 750H105Q132 732 159 708T220 639T281 542T325 413T343 251T325 89T281 -40T221 -138T159 -207T105 -249H90Q80 -249 76 -249T68 -245T64 -234Q64 -230 81 -212T123 -160T172 -75T214 61T231 251Z"></path><path stroke-width="0" id="E272-MJMAINB-53" d="M64 493Q64 582 120 636T264 696H272Q280 697 285 697Q380 697 454 645L480 669Q484 672 488 676T495 683T500 688T504 691T508 693T511 695T514 696T517 697T522 697Q536 697 539 691T542 652V577Q542 557 542 532T543 500Q543 472 540 465T524 458H511H505Q489 458 485 461T479 478Q472 529 449 564T393 614T336 634T287 639Q228 639 203 610T177 544Q177 517 195 493T247 457Q253 454 343 436T475 391Q574 326 574 207V200Q574 163 559 120Q517 12 389 -9Q380 -10 346 -10Q308 -10 275 -5T221 7T184 22T160 35T151 40L126 17Q122 14 118 10T111 3T106 -2T102 -5T98 -7T95 -9T92 -10T89 -11T84 -11Q70 -11 67 -4T64 35V108Q64 128 64 153T63 185Q63 203 63 211T69 223T77 227T94 228H100Q118 228 122 225T126 205Q130 125 193 88T345 51Q408 51 434 82T460 157Q460 196 439 221T388 257Q384 259 305 276T221 295Q155 313 110 366T64 493Z"></path><path stroke-width="0" id="E272-MJMAINB-41" d="M296 0Q278 3 164 3Q58 3 49 0H40V62H92Q144 62 144 64Q388 682 397 689Q403 698 434 698Q463 698 471 689Q475 686 538 530T663 218L724 64Q724 62 776 62H828V0H817Q796 3 658 3Q509 3 485 0H472V62H517Q561 62 561 63L517 175H262L240 120Q218 65 217 64Q217 62 261 62H306V0H296ZM390 237L492 238L440 365Q390 491 388 491Q287 239 287 237H390Z"></path><path stroke-width="0" id="E272-MJMAINB-52" d="M394 0Q370 3 222 3Q75 3 51 0H39V62H147V624H39V686H234Q256 686 299 686T362 687Q479 687 554 669T681 593Q716 550 716 497Q716 390 568 338Q569 337 572 336T577 332Q605 317 623 300T650 258T662 218T668 172Q678 98 689 76Q707 40 748 40Q770 40 780 54T795 88T801 111Q805 117 827 117H831Q846 117 852 113T858 92Q857 78 852 63T834 30T797 1T739 -11Q630 -11 580 12T511 87Q506 104 506 168Q506 170 506 178T507 194Q507 289 438 313Q424 318 356 318H298V62H406V0H394ZM366 369Q459 370 490 381Q548 402 548 476V498V517Q548 578 513 600Q479 624 392 624H358H298V369H366Z"></path><path stroke-width="0" id="E272-MJMAIN-22EF" d="M78 250Q78 274 95 292T138 310Q162 310 180 294T199 251Q199 226 182 208T139 190T96 207T78 250ZM525 250Q525 274 542 292T585 310Q609 310 627 294T646 251Q646 226 629 208T586 190T543 207T525 250ZM972 250Q972 274 989 292T1032 310Q1056 310 1074 294T1093 251Q1093 226 1076 208T1033 190T990 207T972 250Z"></path><path stroke-width="0" id="E272-MJMAIN-36" d="M42 313Q42 476 123 571T303 666Q372 666 402 630T432 550Q432 525 418 510T379 495Q356 495 341 509T326 548Q326 592 373 601Q351 623 311 626Q240 626 194 566Q147 500 147 364L148 360Q153 366 156 373Q197 433 263 433H267Q313 433 348 414Q372 400 396 374T435 317Q456 268 456 210V192Q456 169 451 149Q440 90 387 34T253 -22Q225 -22 199 -14T143 16T92 75T56 172T42 313ZM257 397Q227 397 205 380T171 335T154 278T148 216Q148 133 160 97T198 39Q222 21 251 21Q302 21 329 59Q342 77 347 104T352 209Q352 289 347 316T329 361Q302 397 257 397Z"></path><path stroke-width="0" id="E272-MJMAIN-2D" d="M11 179V252H277V179H11Z"></path><path stroke-width="0" id="E272-MJMAIN-33" d="M127 463Q100 463 85 480T69 524Q69 579 117 622T233 665Q268 665 277 664Q351 652 390 611T430 522Q430 470 396 421T302 350L299 348Q299 347 308 345T337 336T375 315Q457 262 457 175Q457 96 395 37T238 -22Q158 -22 100 21T42 130Q42 158 60 175T105 193Q133 193 151 175T169 130Q169 119 166 110T159 94T148 82T136 74T126 70T118 67L114 66Q165 21 238 21Q293 21 321 74Q338 107 338 175V195Q338 290 274 322Q259 328 213 329L171 330L168 332Q166 335 166 348Q166 366 174 366Q202 366 232 371Q266 376 294 413T322 525V533Q322 590 287 612Q265 626 240 626Q208 626 181 615T143 592T132 580H135Q138 579 143 578T153 573T165 566T175 555T183 540T186 520Q186 498 172 481T127 463Z"></path><path stroke-width="0" id="E272-MJMAIN-32" d="M109 429Q82 429 66 447T50 491Q50 562 103 614T235 666Q326 666 387 610T449 465Q449 422 429 383T381 315T301 241Q265 210 201 149L142 93L218 92Q375 92 385 97Q392 99 409 186V189H449V186Q448 183 436 95T421 3V0H50V19V31Q50 38 56 46T86 81Q115 113 136 137Q145 147 170 174T204 211T233 244T261 278T284 308T305 340T320 369T333 401T340 431T343 464Q343 527 309 573T212 619Q179 619 154 602T119 569T109 550Q109 549 114 549Q132 549 151 535T170 489Q170 464 154 447T109 429Z"></path><path stroke-width="0" id="E272-MJMAIN-2E" d="M78 60Q78 84 95 102T138 120Q162 120 180 104T199 61Q199 36 182 18T139 0T96 17T78 60Z"></path><path stroke-width="0" id="E272-MJMAIN-34" d="M462 0Q444 3 333 3Q217 3 199 0H190V46H221Q241 46 248 46T265 48T279 53T286 61Q287 63 287 115V165H28V211L179 442Q332 674 334 675Q336 677 355 677H373L379 671V211H471V165H379V114Q379 73 379 66T385 54Q393 47 442 46H471V0H462ZM293 211V545L74 212L183 211H293Z"></path><path stroke-width="0" id="E272-MJMAINB-7A" d="M48 262Q48 264 54 349T60 436V444H252Q289 444 336 444T394 445Q441 445 450 441T459 418Q459 406 458 404Q456 399 327 229T194 55H237Q260 56 268 56T297 58T325 65T348 77T370 98T384 128T395 170Q400 197 400 216Q400 217 431 217H462V211Q461 208 453 108T444 6V0H245Q46 0 43 2Q32 7 32 28V33Q32 41 40 52T84 112Q129 170 164 217L298 393H256Q189 392 165 380Q124 360 115 303Q110 280 110 256Q110 254 79 254H48V262Z"></path><path stroke-width="0" id="E272-MJMAIN-2190" d="M944 261T944 250T929 230H165Q167 228 182 216T211 189T244 152T277 96T303 25Q308 7 308 0Q308 -11 288 -11Q281 -11 278 -11T272 -7T267 2T263 21Q245 94 195 151T73 236Q58 242 55 247Q55 254 59 257T73 264Q121 283 158 314T215 375T247 434T264 480L267 497Q269 503 270 505T275 509T288 511Q308 511 308 500Q308 493 303 475Q293 438 278 406T246 352T215 315T185 287T165 270H929Q944 261 944 250Z"></path><path stroke-width="0" id="E272-MJMATHI-3B3" d="M31 249Q11 249 11 258Q11 275 26 304T66 365T129 418T206 441Q233 441 239 440Q287 429 318 386T371 255Q385 195 385 170Q385 166 386 166L398 193Q418 244 443 300T486 391T508 430Q510 431 524 431H537Q543 425 543 422Q543 418 522 378T463 251T391 71Q385 55 378 6T357 -100Q341 -165 330 -190T303 -216Q286 -216 286 -188Q286 -138 340 32L346 51L347 69Q348 79 348 100Q348 257 291 317Q251 355 196 355Q148 355 108 329T51 260Q49 251 47 251Q45 249 31 249Z"></path><path stroke-width="0" id="E272-MJMAIN-2B" d="M56 237T56 250T70 270H369V420L370 570Q380 583 389 583Q402 583 409 568V270H707Q722 262 722 250T707 230H409V-68Q401 -82 391 -82H389H387Q375 -82 369 -68V230H70Q56 237 56 250Z"></path><path stroke-width="0" id="E272-MJMAIN-2207" d="M46 676Q46 679 51 683H781Q786 679 786 676Q786 674 617 326T444 -26Q439 -33 416 -33T388 -26Q385 -22 216 326T46 676ZM697 596Q697 597 445 597T193 596Q195 591 319 336T445 80L697 596Z"></path><path stroke-width="0" id="E272-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E272-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E272-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E272-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E272-MJMATHI-41" d="M208 74Q208 50 254 46Q272 46 272 35Q272 34 270 22Q267 8 264 4T251 0Q249 0 239 0T205 1T141 2Q70 2 50 0H42Q35 7 35 11Q37 38 48 46H62Q132 49 164 96Q170 102 345 401T523 704Q530 716 547 716H555H572Q578 707 578 706L606 383Q634 60 636 57Q641 46 701 46Q726 46 726 36Q726 34 723 22Q720 7 718 4T704 0Q701 0 690 0T651 1T578 2Q484 2 455 0H443Q437 6 437 9T439 27Q443 40 445 43L449 46H469Q523 49 533 63L521 213H283L249 155Q208 86 208 74ZM516 260Q516 271 504 416T490 562L463 519Q447 492 400 412L310 260L413 259Q516 259 516 260Z"></path><path stroke-width="0" id="E272-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E272-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E272-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E272-MJMAIN-35" d="M164 157Q164 133 148 117T109 101H102Q148 22 224 22Q294 22 326 82Q345 115 345 210Q345 313 318 349Q292 382 260 382H254Q176 382 136 314Q132 307 129 306T114 304Q97 304 95 310Q93 314 93 485V614Q93 664 98 664Q100 666 102 666Q103 666 123 658T178 642T253 634Q324 634 389 662Q397 666 402 666Q410 666 410 648V635Q328 538 205 538Q174 538 149 544L139 546V374Q158 388 169 396T205 412T256 420Q337 420 393 355T449 201Q449 109 385 44T229 -22Q148 -22 99 32T50 154Q50 178 61 192T84 210T107 214Q132 214 148 197T164 157Z"></path><path stroke-width="0" id="E272-MJMATHI-3B1" d="M34 156Q34 270 120 356T309 442Q379 442 421 402T478 304Q484 275 485 237V208Q534 282 560 374Q564 388 566 390T582 393Q603 393 603 385Q603 376 594 346T558 261T497 161L486 147L487 123Q489 67 495 47T514 26Q528 28 540 37T557 60Q559 67 562 68T577 70Q597 70 597 62Q597 56 591 43Q579 19 556 5T512 -10H505Q438 -10 414 62L411 69L400 61Q390 53 370 41T325 18T267 -2T203 -11Q124 -11 79 39T34 156ZM208 26Q257 26 306 47T379 90L403 112Q401 255 396 290Q382 405 304 405Q235 405 183 332Q156 292 139 224T121 120Q121 71 146 49T208 26Z"></path><path stroke-width="0" id="E272-MJMAIN-5B" d="M118 -250V750H255V710H158V-210H255V-250H118Z"></path><path stroke-width="0" id="E272-MJMATHI-55" d="M107 637Q73 637 71 641Q70 643 70 649Q70 673 81 682Q83 683 98 683Q139 681 234 681Q268 681 297 681T342 682T362 682Q378 682 378 672Q378 670 376 658Q371 641 366 638H364Q362 638 359 638T352 638T343 637T334 637Q295 636 284 634T266 623Q265 621 238 518T184 302T154 169Q152 155 152 140Q152 86 183 55T269 24Q336 24 403 69T501 205L552 406Q599 598 599 606Q599 633 535 637Q511 637 511 648Q511 650 513 660Q517 676 519 679T529 683Q532 683 561 682T645 680Q696 680 723 681T752 682Q767 682 767 672Q767 650 759 642Q756 637 737 637Q666 633 648 597Q646 592 598 404Q557 235 548 205Q515 105 433 42T263 -22Q171 -22 116 34T60 167V183Q60 201 115 421Q164 622 164 628Q164 635 107 637Z"></path><path stroke-width="0" id="E272-MJMAIN-2212" d="M84 237T84 250T98 270H679Q694 262 694 250T679 230H98Q84 237 84 250Z"></path><path stroke-width="0" id="E272-MJMAIN-5D" d="M22 710V750H159V-250H22V-210H119V710H22Z"></path><path stroke-width="0" id="E272-MJMAIN-2032" d="M79 43Q73 43 52 49T30 61Q30 68 85 293T146 528Q161 560 198 560Q218 560 240 545T262 501Q262 496 260 486Q259 479 173 263T84 45T79 43Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><g transform="translate(7432,-2532)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">算</text><g transform="translate(1052,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">法</text></g><use transform="scale(1.2)" xlink:href="#E272-MJMAINB-36" x="1963" y="0"></use><use transform="scale(1.2)" xlink:href="#E272-MJMAINB-2D" x="2538" y="0"></use><use transform="scale(1.2)" xlink:href="#E272-MJMAINB-35" x="2921" y="0"></use><use transform="scale(1.2)" xlink:href="#E272-MJMAINB-54" x="4121" y="0"></use><use transform="scale(1.2)" xlink:href="#E272-MJMAINB-44" x="4921" y="0"></use><use transform="scale(1.2)" xlink:href="#E272-MJMAINB-28" x="5803" y="0"></use><use transform="scale(1.2)" xlink:href="#E272-MJMATHI-3BB" x="6250" y="0"></use><g transform="translate(8199,0)"><use transform="scale(1.2)" xlink:href="#E272-MJMAINB-29"></use><g transform="translate(786,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">算</text></g><g transform="translate(1839,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">法</text></g><g transform="translate(2892,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">估</text></g><g transform="translate(3944,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">计</text></g><g transform="translate(4997,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">动</text></g><g transform="translate(6014,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">作</text></g><g transform="translate(7066,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">价</text></g><g transform="translate(8119,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">值</text></g><g transform="translate(9172,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">或</text></g><use transform="scale(1.2)" xlink:href="#E272-MJMAINB-53" x="8729" y="0"></use><use transform="scale(1.2)" xlink:href="#E272-MJMAINB-41" x="9368" y="0"></use><use transform="scale(1.2)" xlink:href="#E272-MJMAINB-52" x="10237" y="0"></use><use transform="scale(1.2)" xlink:href="#E272-MJMAINB-53" x="11099" y="0"></use><use transform="scale(1.2)" xlink:href="#E272-MJMAINB-41" x="11738" y="0"></use><use transform="scale(1.2)" xlink:href="#E272-MJMAINB-28" x="12607" y="0"></use></g><use transform="scale(1.2)" xlink:href="#E272-MJMATHI-3BB" x="19887" y="0"></use><g transform="translate(24564,0)"><use transform="scale(1.2)" xlink:href="#E272-MJMAINB-29"></use><g transform="translate(786,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">算</text></g><g transform="translate(1839,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">法</text></g></g></g><g transform="translate(0,-7466)"><g transform="translate(-19,0)"><g transform="translate(0,3404)"><g><rect fill="black" stroke="none" width="1569" height="100" x="0" y="500"></rect></g></g><g transform="translate(0,-3205)"><g><rect fill="black" stroke="none" width="1569" height="100" x="0" y="-500"></rect></g></g></g><g transform="translate(1551,0)"><g transform="translate(0,3404)"><g><rect fill="black" stroke="none" width="41597" height="100" x="0" y="500"></rect></g></g><g transform="translate(0,2104)"><use xlink:href="#E272-MJMAIN-22EF" x="166" y="0"></use><g transform="translate(2505,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">同</text><g transform="translate(830,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">算</text></g><g transform="translate(1661,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">法</text></g><use xlink:href="#E272-MJMAIN-36" x="2741" y="0"></use><use xlink:href="#E272-MJMAIN-2D" x="3241" y="0"></use><use xlink:href="#E272-MJMAIN-33" x="3574" y="0"></use></g><use xlink:href="#E272-MJMAIN-22EF" x="7746" y="0"></use></g><g transform="translate(0,804)"><use xlink:href="#E272-MJMAIN-32"></use><use xlink:href="#E272-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E272-MJMAIN-32" x="778" y="0"></use><use xlink:href="#E272-MJMAIN-2E" x="1278" y="0"></use><use xlink:href="#E272-MJMAIN-34" x="1556" y="0"></use><g transform="translate(2056,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(2886,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">更</text></g><g transform="translate(3717,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">新</text></g><g transform="translate(4547,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">资</text></g><g transform="translate(5378,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">格</text></g><g transform="translate(6209,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">迹</text></g><g transform="translate(7039,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(7870,0)"><use xlink:href="#E272-MJMAINB-7A" x="0" y="0"></use><use xlink:href="#E272-MJMAIN-2190" x="788" y="0"></use><use xlink:href="#E272-MJMATHI-3B3" x="2066" y="0"></use><use xlink:href="#E272-MJMATHI-3BB" x="2609" y="0"></use><use xlink:href="#E272-MJMAINB-7A" x="3192" y="0"></use><use xlink:href="#E272-MJMAIN-2B" x="3925" y="0"></use><use xlink:href="#E272-MJMAIN-2207" x="4926" y="0"></use><use xlink:href="#E272-MJMATHI-71" x="5759" y="0"></use><use xlink:href="#E272-MJMAIN-28" x="6219" y="0"></use><use xlink:href="#E272-MJMATHI-53" x="6608" y="0"></use><use xlink:href="#E272-MJMAIN-2C" x="7253" y="0"></use><use xlink:href="#E272-MJMATHI-41" x="7697" y="0"></use><use xlink:href="#E272-MJMAIN-3B" x="8447" y="0"></use><use xlink:href="#E272-MJMAINB-77" x="8892" y="0"></use><use xlink:href="#E272-MJMAIN-29" x="9723" y="0"></use></g><g transform="translate(17982,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">；</text></g></g></g><g transform="translate(0,-546)"><use xlink:href="#E272-MJMAIN-32"></use><use xlink:href="#E272-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E272-MJMAIN-32" x="778" y="0"></use><use xlink:href="#E272-MJMAIN-2E" x="1278" y="0"></use><use xlink:href="#E272-MJMAIN-35" x="1556" y="0"></use><g transform="translate(2056,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(2886,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">更</text></g><g transform="translate(3717,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">新</text></g><g transform="translate(4547,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">价</text></g><g transform="translate(5378,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">值</text></g><g transform="translate(6209,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(7039,0)"><use xlink:href="#E272-MJMAINB-77" x="0" y="0"></use><use xlink:href="#E272-MJMAIN-2190" x="1108" y="0"></use><use xlink:href="#E272-MJMAINB-77" x="2386" y="0"></use><use xlink:href="#E272-MJMAIN-2B" x="3439" y="0"></use><use xlink:href="#E272-MJMATHI-3B1" x="4440" y="0"></use><use xlink:href="#E272-MJMAIN-5B" x="5080" y="0"></use><use xlink:href="#E272-MJMATHI-55" x="5358" y="0"></use><use xlink:href="#E272-MJMAIN-2212" x="6347" y="0"></use><use xlink:href="#E272-MJMATHI-71" x="7347" y="0"></use><use xlink:href="#E272-MJMAIN-28" x="7807" y="0"></use><use xlink:href="#E272-MJMATHI-53" x="8196" y="0"></use><use xlink:href="#E272-MJMAIN-2C" x="8841" y="0"></use><use xlink:href="#E272-MJMATHI-41" x="9286" y="0"></use><use xlink:href="#E272-MJMAIN-3B" x="10036" y="0"></use><use xlink:href="#E272-MJMAINB-77" x="10480" y="0"></use><use xlink:href="#E272-MJMAIN-29" x="11311" y="0"></use><use xlink:href="#E272-MJMAIN-5D" x="11700" y="0"></use><use xlink:href="#E272-MJMAINB-7A" x="11978" y="0"></use></g><g transform="translate(19529,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">；</text></g></g></g><g transform="translate(0,-1905)"><use xlink:href="#E272-MJMAIN-32"></use><use xlink:href="#E272-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E272-MJMAIN-32" x="778" y="0"></use><use xlink:href="#E272-MJMAIN-2E" x="1278" y="0"></use><use xlink:href="#E272-MJMAIN-36" x="1556" y="0"></use><g transform="translate(2306,0)"><use xlink:href="#E272-MJMATHI-53" x="444" y="0"></use><use xlink:href="#E272-MJMAIN-2190" x="1367" y="0"></use><g transform="translate(2645,0)"><use xlink:href="#E272-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E272-MJMAIN-2032" x="925" y="583"></use></g><use xlink:href="#E272-MJMAIN-2C" x="3594" y="0"></use><use xlink:href="#E272-MJMATHI-41" x="4316" y="0"></use><use xlink:href="#E272-MJMAIN-2190" x="5344" y="0"></use><g transform="translate(6622,0)"><use xlink:href="#E272-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E272-MJMAIN-2032" x="1060" y="583"></use></g></g><g transform="translate(9972,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">。</text></g></g></g><g transform="translate(0,-3205)"><g><rect fill="black" stroke="none" width="41597" height="100" x="0" y="-500"></rect></g></g></g></g></g></svg></span></div><script type="math/tex; mode=display" id="MathJax-Element-245">\; \\ \; \\
\large \textbf{算法 6-5   TD($\lambda$) 算法估计动作价值或 SARSA($\lambda$) 算法} \\
\begin{split}
\rule[5pt]{10mm}{0.1em} &\rule[5pt]{265mm}{0.1em} \\
&\cdots \quad \text{同算法 6-3} \quad \cdots \\
&\text{2.2.4（更新资格迹）$\bold z \leftarrow \gamma\lambda\bold z + \nabla q(S,A;\bold w)$ ；} \\
&\text{2.2.5（更新价值）$\bold w \leftarrow \bold w + \alpha[U-q(S,A;\bold w)]\bold z$ ；} \\
&\text{2.2.6 $\;\, S \leftarrow S',\; A \leftarrow A'$ 。}\\
\rule[-5pt]{10mm}{0.1em} &\rule[-5pt]{265mm}{0.1em}
\end{split}
\; \\ \; \\</script></div></div><div contenteditable="false" spellcheck="false" class="mathjax-block md-end-block md-math-block md-rawblock" id="mathjax-n24" cid="n24" mdtype="math_block"><div class="md-rawblock-container md-math-container" tabindex="-1"><div class="MathJax_SVG_Display"><span class="MathJax_SVG" id="MathJax-Element-246-Frame" tabindex="-1" style="font-size: 100%; display: inline-block; zoom: 0.971345;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="101.294ex" height="42.26ex" viewBox="-18.1 -43.5 43612.5 18195.3" role="img" focusable="false" style="vertical-align: -42.159ex; margin-left: -0.042ex; max-width: 100%;"><defs><path stroke-width="0" id="E273-MJMAINB-36" d="M48 318Q48 395 68 456T120 553T193 613T273 646T350 655Q425 655 461 616T497 524Q497 485 475 468T428 451Q399 451 378 470T357 521Q357 565 403 588Q375 601 351 601Q313 601 282 584Q242 565 222 526Q199 473 199 367Q201 369 210 380T227 396T246 410T275 422T312 426Q438 426 494 332Q526 285 526 208V199Q526 112 465 53Q428 17 388 3T285 -11Q236 -11 195 7T135 43T104 80Q48 165 48 318ZM375 231V244V268Q375 295 373 310T364 342T341 366T299 374H297Q231 374 208 287Q200 257 200 196Q201 120 209 100Q231 47 288 47Q351 47 368 90Q375 112 375 231Z"></path><path stroke-width="0" id="E273-MJMAINB-2D" d="M13 166V278H318V166H13Z"></path><path stroke-width="0" id="E273-MJMAINB-54" d="M41 425Q41 426 51 545T62 669V675H737V669Q738 665 748 546T758 425V419H696V425Q687 517 669 555T595 607Q578 612 522 613H478V62H631V0H615Q585 3 399 3Q214 3 184 0H168V62H321V613H277H263Q164 613 134 561Q113 527 103 425V419H41V425Z"></path><path stroke-width="0" id="E273-MJMAINB-44" d="M39 624V686H270H310H408Q500 686 545 680T638 649Q768 584 805 438Q817 388 817 338Q817 171 702 75Q628 17 515 2Q504 1 270 0H39V62H147V624H39ZM655 337Q655 370 655 390T650 442T639 494T616 540T580 580T526 607T451 623Q443 624 368 624H298V62H377H387H407Q445 62 472 65T540 83T606 129Q629 156 640 195T653 262T655 337Z"></path><path stroke-width="0" id="E273-MJMAINB-28" d="M103 166T103 251T121 412T165 541T225 639T287 708T341 750H356H361Q382 750 382 736Q382 732 365 714T323 661T274 576T232 439T214 250Q214 -62 381 -229Q382 -231 382 -234Q382 -249 360 -249H356H341Q314 -231 287 -207T226 -138T165 -41T121 89Z"></path><path stroke-width="0" id="E273-MJMATHI-3BB" d="M166 673Q166 685 183 694H202Q292 691 316 644Q322 629 373 486T474 207T524 67Q531 47 537 34T546 15T551 6T555 2T556 -2T550 -11H482Q457 3 450 18T399 152L354 277L340 262Q327 246 293 207T236 141Q211 112 174 69Q123 9 111 -1T83 -12Q47 -12 47 20Q47 37 61 52T199 187Q229 216 266 252T321 306L338 322Q338 323 288 462T234 612Q214 657 183 657Q166 657 166 673Z"></path><path stroke-width="0" id="E273-MJMAINB-29" d="M231 251Q231 354 214 439T173 575T123 661T81 714T64 735Q64 744 73 749H75Q77 749 79 749T84 750T90 750H105Q132 732 159 708T220 639T281 542T325 413T343 251T325 89T281 -40T221 -138T159 -207T105 -249H90Q80 -249 76 -249T68 -245T64 -234Q64 -230 81 -212T123 -160T172 -75T214 61T231 251Z"></path><path stroke-width="0" id="E273-MJMAINB-53" d="M64 493Q64 582 120 636T264 696H272Q280 697 285 697Q380 697 454 645L480 669Q484 672 488 676T495 683T500 688T504 691T508 693T511 695T514 696T517 697T522 697Q536 697 539 691T542 652V577Q542 557 542 532T543 500Q543 472 540 465T524 458H511H505Q489 458 485 461T479 478Q472 529 449 564T393 614T336 634T287 639Q228 639 203 610T177 544Q177 517 195 493T247 457Q253 454 343 436T475 391Q574 326 574 207V200Q574 163 559 120Q517 12 389 -9Q380 -10 346 -10Q308 -10 275 -5T221 7T184 22T160 35T151 40L126 17Q122 14 118 10T111 3T106 -2T102 -5T98 -7T95 -9T92 -10T89 -11T84 -11Q70 -11 67 -4T64 35V108Q64 128 64 153T63 185Q63 203 63 211T69 223T77 227T94 228H100Q118 228 122 225T126 205Q130 125 193 88T345 51Q408 51 434 82T460 157Q460 196 439 221T388 257Q384 259 305 276T221 295Q155 313 110 366T64 493Z"></path><path stroke-width="0" id="E273-MJMAINB-41" d="M296 0Q278 3 164 3Q58 3 49 0H40V62H92Q144 62 144 64Q388 682 397 689Q403 698 434 698Q463 698 471 689Q475 686 538 530T663 218L724 64Q724 62 776 62H828V0H817Q796 3 658 3Q509 3 485 0H472V62H517Q561 62 561 63L517 175H262L240 120Q218 65 217 64Q217 62 261 62H306V0H296ZM390 237L492 238L440 365Q390 491 388 491Q287 239 287 237H390Z"></path><path stroke-width="0" id="E273-MJMAINB-52" d="M394 0Q370 3 222 3Q75 3 51 0H39V62H147V624H39V686H234Q256 686 299 686T362 687Q479 687 554 669T681 593Q716 550 716 497Q716 390 568 338Q569 337 572 336T577 332Q605 317 623 300T650 258T662 218T668 172Q678 98 689 76Q707 40 748 40Q770 40 780 54T795 88T801 111Q805 117 827 117H831Q846 117 852 113T858 92Q857 78 852 63T834 30T797 1T739 -11Q630 -11 580 12T511 87Q506 104 506 168Q506 170 506 178T507 194Q507 289 438 313Q424 318 356 318H298V62H406V0H394ZM366 369Q459 370 490 381Q548 402 548 476V498V517Q548 578 513 600Q479 624 392 624H358H298V369H366Z"></path><path stroke-width="0" id="E273-MJMAINB-51" d="M64 339Q64 431 96 502T182 614T295 675T420 696Q469 696 481 695Q620 680 709 589T798 339Q798 255 768 184Q720 77 611 26L600 21Q635 -26 682 -26H696Q769 -26 769 0Q769 7 774 12T787 18Q805 18 805 -7V-13Q803 -64 785 -106T737 -171Q720 -183 697 -191Q687 -193 668 -193Q636 -193 613 -182T575 -144T552 -94T532 -27Q531 -23 530 -16T528 -6T526 -3L512 -5Q499 -7 477 -8T431 -10Q393 -10 382 -9Q238 8 151 97T64 339ZM326 80Q326 113 356 138T430 163Q492 163 542 100L553 86Q554 85 561 91T578 108Q637 179 637 330Q637 430 619 498T548 604Q500 641 425 641Q408 641 390 637T347 623T299 590T259 535Q226 469 226 338Q226 244 246 180T318 79L325 74Q326 74 326 80ZM506 58Q480 112 433 112Q412 112 395 104T378 77Q378 44 431 44Q480 44 506 58Z"></path><path stroke-width="0" id="E273-MJMAIN-22EF" d="M78 250Q78 274 95 292T138 310Q162 310 180 294T199 251Q199 226 182 208T139 190T96 207T78 250ZM525 250Q525 274 542 292T585 310Q609 310 627 294T646 251Q646 226 629 208T586 190T543 207T525 250ZM972 250Q972 274 989 292T1032 310Q1056 310 1074 294T1093 251Q1093 226 1076 208T1033 190T990 207T972 250Z"></path><path stroke-width="0" id="E273-MJMAIN-36" d="M42 313Q42 476 123 571T303 666Q372 666 402 630T432 550Q432 525 418 510T379 495Q356 495 341 509T326 548Q326 592 373 601Q351 623 311 626Q240 626 194 566Q147 500 147 364L148 360Q153 366 156 373Q197 433 263 433H267Q313 433 348 414Q372 400 396 374T435 317Q456 268 456 210V192Q456 169 451 149Q440 90 387 34T253 -22Q225 -22 199 -14T143 16T92 75T56 172T42 313ZM257 397Q227 397 205 380T171 335T154 278T148 216Q148 133 160 97T198 39Q222 21 251 21Q302 21 329 59Q342 77 347 104T352 209Q352 289 347 316T329 361Q302 397 257 397Z"></path><path stroke-width="0" id="E273-MJMAIN-2D" d="M11 179V252H277V179H11Z"></path><path stroke-width="0" id="E273-MJMAIN-34" d="M462 0Q444 3 333 3Q217 3 199 0H190V46H221Q241 46 248 46T265 48T279 53T286 61Q287 63 287 115V165H28V211L179 442Q332 674 334 675Q336 677 355 677H373L379 671V211H471V165H379V114Q379 73 379 66T385 54Q393 47 442 46H471V0H462ZM293 211V545L74 212L183 211H293Z"></path><path stroke-width="0" id="E273-MJMAIN-32" d="M109 429Q82 429 66 447T50 491Q50 562 103 614T235 666Q326 666 387 610T449 465Q449 422 429 383T381 315T301 241Q265 210 201 149L142 93L218 92Q375 92 385 97Q392 99 409 186V189H449V186Q448 183 436 95T421 3V0H50V19V31Q50 38 56 46T86 81Q115 113 136 137Q145 147 170 174T204 211T233 244T261 278T284 308T305 340T320 369T333 401T340 431T343 464Q343 527 309 573T212 619Q179 619 154 602T119 569T109 550Q109 549 114 549Q132 549 151 535T170 489Q170 464 154 447T109 429Z"></path><path stroke-width="0" id="E273-MJMAIN-2E" d="M78 60Q78 84 95 102T138 120Q162 120 180 104T199 61Q199 36 182 18T139 0T96 17T78 60Z"></path><path stroke-width="0" id="E273-MJMAIN-31" d="M213 578L200 573Q186 568 160 563T102 556H83V602H102Q149 604 189 617T245 641T273 663Q275 666 285 666Q294 666 302 660V361L303 61Q310 54 315 52T339 48T401 46H427V0H416Q395 3 257 3Q121 3 100 0H88V46H114Q136 46 152 46T177 47T193 50T201 52T207 57T213 61V578Z"></path><path stroke-width="0" id="E273-MJMAINB-7A" d="M48 262Q48 264 54 349T60 436V444H252Q289 444 336 444T394 445Q441 445 450 441T459 418Q459 406 458 404Q456 399 327 229T194 55H237Q260 56 268 56T297 58T325 65T348 77T370 98T384 128T395 170Q400 197 400 216Q400 217 431 217H462V211Q461 208 453 108T444 6V0H245Q46 0 43 2Q32 7 32 28V33Q32 41 40 52T84 112Q129 170 164 217L298 393H256Q189 392 165 380Q124 360 115 303Q110 280 110 256Q110 254 79 254H48V262Z"></path><path stroke-width="0" id="E273-MJMAIN-2190" d="M944 261T944 250T929 230H165Q167 228 182 216T211 189T244 152T277 96T303 25Q308 7 308 0Q308 -11 288 -11Q281 -11 278 -11T272 -7T267 2T263 21Q245 94 195 151T73 236Q58 242 55 247Q55 254 59 257T73 264Q121 283 158 314T215 375T247 434T264 480L267 497Q269 503 270 505T275 509T288 511Q308 511 308 500Q308 493 303 475Q293 438 278 406T246 352T215 315T185 287T165 270H929Q944 261 944 250Z"></path><path stroke-width="0" id="E273-MJMAINB-30" d="M266 654H280H282Q500 654 524 418Q529 370 529 320Q529 125 456 52Q397 -10 287 -10Q110 -10 63 154Q45 212 45 316Q45 504 113 585Q140 618 185 636T266 654ZM374 548Q347 604 286 604Q247 604 218 575Q197 552 193 511T188 311Q188 159 196 116Q202 87 225 64T287 41Q339 41 367 87Q379 107 382 152T386 329Q386 518 374 548Z"></path><path stroke-width="0" id="E273-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E273-MJMAIN-35" d="M164 157Q164 133 148 117T109 101H102Q148 22 224 22Q294 22 326 82Q345 115 345 210Q345 313 318 349Q292 382 260 382H254Q176 382 136 314Q132 307 129 306T114 304Q97 304 95 310Q93 314 93 485V614Q93 664 98 664Q100 666 102 666Q103 666 123 658T178 642T253 634Q324 634 389 662Q397 666 402 666Q410 666 410 648V635Q328 538 205 538Q174 538 149 544L139 546V374Q158 388 169 396T205 412T256 420Q337 420 393 355T449 201Q449 109 385 44T229 -22Q148 -22 99 32T50 154Q50 178 61 192T84 210T107 214Q132 214 148 197T164 157Z"></path><path stroke-width="0" id="E273-MJMATHI-3B3" d="M31 249Q11 249 11 258Q11 275 26 304T66 365T129 418T206 441Q233 441 239 440Q287 429 318 386T371 255Q385 195 385 170Q385 166 386 166L398 193Q418 244 443 300T486 391T508 430Q510 431 524 431H537Q543 425 543 422Q543 418 522 378T463 251T391 71Q385 55 378 6T357 -100Q341 -165 330 -190T303 -216Q286 -216 286 -188Q286 -138 340 32L346 51L347 69Q348 79 348 100Q348 257 291 317Q251 355 196 355Q148 355 108 329T51 260Q49 251 47 251Q45 249 31 249Z"></path><path stroke-width="0" id="E273-MJMAIN-2B" d="M56 237T56 250T70 270H369V420L370 570Q380 583 389 583Q402 583 409 568V270H707Q722 262 722 250T707 230H409V-68Q401 -82 391 -82H389H387Q375 -82 369 -68V230H70Q56 237 56 250Z"></path><path stroke-width="0" id="E273-MJMAIN-2207" d="M46 676Q46 679 51 683H781Q786 679 786 676Q786 674 617 326T444 -26Q439 -33 416 -33T388 -26Q385 -22 216 326T46 676ZM697 596Q697 597 445 597T193 596Q195 591 319 336T445 80L697 596Z"></path><path stroke-width="0" id="E273-MJMATHI-76" d="M173 380Q173 405 154 405Q130 405 104 376T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Q21 294 29 316T53 368T97 419T160 441Q202 441 225 417T249 361Q249 344 246 335Q246 329 231 291T200 202T182 113Q182 86 187 69Q200 26 250 26Q287 26 319 60T369 139T398 222T409 277Q409 300 401 317T383 343T365 361T357 383Q357 405 376 424T417 443Q436 443 451 425T467 367Q467 340 455 284T418 159T347 40T241 -11Q177 -11 139 22Q102 54 102 117Q102 148 110 181T151 298Q173 362 173 380Z"></path><path stroke-width="0" id="E273-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E273-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E273-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E273-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E273-MJMAIN-53" d="M55 507Q55 590 112 647T243 704H257Q342 704 405 641L426 672Q431 679 436 687T446 700L449 704Q450 704 453 704T459 705H463Q466 705 472 699V462L466 456H448Q437 456 435 459T430 479Q413 605 329 646Q292 662 254 662Q201 662 168 626T135 542Q135 508 152 480T200 435Q210 431 286 412T370 389Q427 367 463 314T500 191Q500 110 448 45T301 -21Q245 -21 201 -4T140 27L122 41Q118 36 107 21T87 -7T78 -21Q76 -22 68 -22H64Q61 -22 55 -16V101Q55 220 56 222Q58 227 76 227H89Q95 221 95 214Q95 182 105 151T139 90T205 42T305 24Q352 24 386 62T420 155Q420 198 398 233T340 281Q284 295 266 300Q261 301 239 306T206 314T174 325T141 343T112 367T85 402Q55 451 55 507Z"></path><path stroke-width="0" id="E273-MJMAIN-41" d="M255 0Q240 3 140 3Q48 3 39 0H32V46H47Q119 49 139 88Q140 91 192 245T295 553T348 708Q351 716 366 716H376Q396 715 400 709Q402 707 508 390L617 67Q624 54 636 51T687 46H717V0H708Q699 3 581 3Q458 3 437 0H427V46H440Q510 46 510 64Q510 66 486 138L462 209H229L209 150Q189 91 189 85Q189 72 209 59T259 46H264V0H255ZM447 255L345 557L244 256Q244 255 345 255H447Z"></path><path stroke-width="0" id="E273-MJMAIN-52" d="M130 622Q123 629 119 631T103 634T60 637H27V683H202H236H300Q376 683 417 677T500 648Q595 600 609 517Q610 512 610 501Q610 468 594 439T556 392T511 361T472 343L456 338Q459 335 467 332Q497 316 516 298T545 254T559 211T568 155T578 94Q588 46 602 31T640 16H645Q660 16 674 32T692 87Q692 98 696 101T712 105T728 103T732 90Q732 59 716 27T672 -16Q656 -22 630 -22Q481 -16 458 90Q456 101 456 163T449 246Q430 304 373 320L363 322L297 323H231V192L232 61Q238 51 249 49T301 46H334V0H323Q302 3 181 3Q59 3 38 0H27V46H60Q102 47 111 49T130 61V622ZM491 499V509Q491 527 490 539T481 570T462 601T424 623T362 636Q360 636 340 636T304 637H283Q238 637 234 628Q231 624 231 492V360H289Q390 360 434 378T489 456Q491 467 491 499Z"></path><path stroke-width="0" id="E273-MJMAIN-51" d="M56 341Q56 499 157 602T388 705Q521 705 621 601T722 341Q722 275 703 218T660 127T603 63T555 25T525 9Q524 8 524 8H523Q524 5 526 -1T537 -21T555 -47T581 -67T615 -76Q653 -76 678 -56T706 -3Q707 10 716 10Q721 10 728 5L727 -13Q727 -88 697 -140T606 -193Q563 -193 538 -166T498 -83Q483 -23 483 -8L471 -11Q459 -14 435 -18T388 -22Q254 -22 155 81T56 341ZM607 339Q607 429 586 496T531 598T461 649T390 665T318 649T248 598T192 496T170 339Q170 143 277 57Q301 39 305 39L304 42Q304 44 304 46Q301 53 301 68Q301 101 325 128T391 155Q454 155 495 70L501 58Q549 91 578 164Q607 234 607 339ZM385 18Q404 18 425 23T459 33T472 40Q471 47 468 57T449 88T412 115Q398 117 386 117Q367 117 353 102T338 67Q338 48 351 33T385 18Z"></path><path stroke-width="0" id="E273-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E273-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E273-MJMATHI-41" d="M208 74Q208 50 254 46Q272 46 272 35Q272 34 270 22Q267 8 264 4T251 0Q249 0 239 0T205 1T141 2Q70 2 50 0H42Q35 7 35 11Q37 38 48 46H62Q132 49 164 96Q170 102 345 401T523 704Q530 716 547 716H555H572Q578 707 578 706L606 383Q634 60 636 57Q641 46 701 46Q726 46 726 36Q726 34 723 22Q720 7 718 4T704 0Q701 0 690 0T651 1T578 2Q484 2 455 0H443Q437 6 437 9T439 27Q443 40 445 43L449 46H469Q523 49 533 63L521 213H283L249 155Q208 86 208 74ZM516 260Q516 271 504 416T490 562L463 519Q447 492 400 412L310 260L413 259Q516 259 516 260Z"></path><path stroke-width="0" id="E273-MJMATHI-3B1" d="M34 156Q34 270 120 356T309 442Q379 442 421 402T478 304Q484 275 485 237V208Q534 282 560 374Q564 388 566 390T582 393Q603 393 603 385Q603 376 594 346T558 261T497 161L486 147L487 123Q489 67 495 47T514 26Q528 28 540 37T557 60Q559 67 562 68T577 70Q597 70 597 62Q597 56 591 43Q579 19 556 5T512 -10H505Q438 -10 414 62L411 69L400 61Q390 53 370 41T325 18T267 -2T203 -11Q124 -11 79 39T34 156ZM208 26Q257 26 306 47T379 90L403 112Q401 255 396 290Q382 405 304 405Q235 405 183 332Q156 292 139 224T121 120Q121 71 146 49T208 26Z"></path><path stroke-width="0" id="E273-MJMAIN-5B" d="M118 -250V750H255V710H158V-210H255V-250H118Z"></path><path stroke-width="0" id="E273-MJMATHI-55" d="M107 637Q73 637 71 641Q70 643 70 649Q70 673 81 682Q83 683 98 683Q139 681 234 681Q268 681 297 681T342 682T362 682Q378 682 378 672Q378 670 376 658Q371 641 366 638H364Q362 638 359 638T352 638T343 637T334 637Q295 636 284 634T266 623Q265 621 238 518T184 302T154 169Q152 155 152 140Q152 86 183 55T269 24Q336 24 403 69T501 205L552 406Q599 598 599 606Q599 633 535 637Q511 637 511 648Q511 650 513 660Q517 676 519 679T529 683Q532 683 561 682T645 680Q696 680 723 681T752 682Q767 682 767 672Q767 650 759 642Q756 637 737 637Q666 633 648 597Q646 592 598 404Q557 235 548 205Q515 105 433 42T263 -22Q171 -22 116 34T60 167V183Q60 201 115 421Q164 622 164 628Q164 635 107 637Z"></path><path stroke-width="0" id="E273-MJMAIN-2212" d="M84 237T84 250T98 270H679Q694 262 694 250T679 230H98Q84 237 84 250Z"></path><path stroke-width="0" id="E273-MJMAIN-5D" d="M22 710V750H159V-250H22V-210H119V710H22Z"></path><path stroke-width="0" id="E273-MJMAIN-2032" d="M79 43Q73 43 52 49T30 61Q30 68 85 293T146 528Q161 560 198 560Q218 560 240 545T262 501Q262 496 260 486Q259 479 173 263T84 45T79 43Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><g transform="translate(4198,-2532)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">算</text><g transform="translate(1052,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">法</text></g><use transform="scale(1.2)" xlink:href="#E273-MJMAINB-36" x="1963" y="0"></use><use transform="scale(1.2)" xlink:href="#E273-MJMAINB-2D" x="2538" y="0"></use><use transform="scale(1.2)" xlink:href="#E273-MJMAINB-36" x="2921" y="0"></use><use transform="scale(1.2)" xlink:href="#E273-MJMAINB-54" x="4121" y="0"></use><use transform="scale(1.2)" xlink:href="#E273-MJMAINB-44" x="4921" y="0"></use><use transform="scale(1.2)" xlink:href="#E273-MJMAINB-28" x="5803" y="0"></use><use transform="scale(1.2)" xlink:href="#E273-MJMATHI-3BB" x="6250" y="0"></use><g transform="translate(8199,0)"><use transform="scale(1.2)" xlink:href="#E273-MJMAINB-29"></use><g transform="translate(786,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">估</text></g><g transform="translate(1839,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">计</text></g><g transform="translate(2892,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">状</text></g><g transform="translate(3944,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">态</text></g><g transform="translate(4997,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">价</text></g><g transform="translate(6050,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">值</text></g><g transform="translate(7103,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">或</text></g><g transform="translate(8156,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">期</text></g><g transform="translate(9209,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">望</text></g><use transform="scale(1.2)" xlink:href="#E273-MJMAINB-53" x="8760" y="0"></use><use transform="scale(1.2)" xlink:href="#E273-MJMAINB-41" x="9399" y="0"></use><use transform="scale(1.2)" xlink:href="#E273-MJMAINB-52" x="10268" y="0"></use><use transform="scale(1.2)" xlink:href="#E273-MJMAINB-53" x="11130" y="0"></use><use transform="scale(1.2)" xlink:href="#E273-MJMAINB-41" x="11769" y="0"></use><use transform="scale(1.2)" xlink:href="#E273-MJMAINB-28" x="12638" y="0"></use></g><use transform="scale(1.2)" xlink:href="#E273-MJMATHI-3BB" x="19918" y="0"></use><g transform="translate(24601,0)"><use transform="scale(1.2)" xlink:href="#E273-MJMAINB-29"></use><g transform="translate(786,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">算</text></g><g transform="translate(1839,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">法</text></g><g transform="translate(2892,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">或</text></g><use transform="scale(1.2)" xlink:href="#E273-MJMAINB-51" x="3495" y="0"></use><use transform="scale(1.2)" xlink:href="#E273-MJMAINB-28" x="4359" y="0"></use></g><use transform="scale(1.2)" xlink:href="#E273-MJMATHI-3BB" x="25307" y="0"></use><g transform="translate(31069,0)"><use transform="scale(1.2)" xlink:href="#E273-MJMAINB-29"></use><g transform="translate(786,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">学</text></g><g transform="translate(1839,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">习</text></g></g></g><g transform="translate(0,-10116)"><g transform="translate(-19,0)"><g transform="translate(0,6054)"><g><rect fill="black" stroke="none" width="1569" height="100" x="0" y="500"></rect></g></g><g transform="translate(0,-5855)"><g><rect fill="black" stroke="none" width="1569" height="100" x="0" y="-500"></rect></g></g></g><g transform="translate(1551,0)"><g transform="translate(0,6054)"><g><rect fill="black" stroke="none" width="41597" height="100" x="0" y="500"></rect></g></g><g transform="translate(0,4754)"><use xlink:href="#E273-MJMAIN-22EF" x="166" y="0"></use><g transform="translate(2505,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">同</text><g transform="translate(830,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">算</text></g><g transform="translate(1661,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">法</text></g><use xlink:href="#E273-MJMAIN-36" x="2741" y="0"></use><use xlink:href="#E273-MJMAIN-2D" x="3241" y="0"></use><use xlink:href="#E273-MJMAIN-34" x="3574" y="0"></use></g><use xlink:href="#E273-MJMAIN-22EF" x="7746" y="0"></use></g><g transform="translate(0,3454)"><use xlink:href="#E273-MJMAIN-32"></use><use xlink:href="#E273-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E273-MJMAIN-31" x="778" y="0"></use><g transform="translate(1278,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(2108,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">初</text></g><g transform="translate(2939,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">始</text></g><g transform="translate(3769,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">化</text></g><g transform="translate(4600,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(5431,0)"><use xlink:href="#E273-MJMAINB-7A" x="0" y="0"></use><use xlink:href="#E273-MJMAIN-2190" x="788" y="0"></use><use xlink:href="#E273-MJMAINB-30" x="2066" y="0"></use></g><g transform="translate(8072,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">，</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">选</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">择</text></g><g transform="translate(2741,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">状</text></g><g transform="translate(3572,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">态</text></g></g><use xlink:href="#E273-MJMATHI-53" x="12726" y="0"></use><g transform="translate(13371,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">。</text></g></g></g><g transform="translate(0,2154)"><use xlink:href="#E273-MJMAIN-22EF" x="166" y="0"></use><g transform="translate(2505,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">同</text><g transform="translate(830,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">算</text></g><g transform="translate(1661,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">法</text></g><use xlink:href="#E273-MJMAIN-36" x="2741" y="0"></use><use xlink:href="#E273-MJMAIN-2D" x="3241" y="0"></use><use xlink:href="#E273-MJMAIN-34" x="3574" y="0"></use></g><use xlink:href="#E273-MJMAIN-22EF" x="7746" y="0"></use></g><g transform="translate(0,854)"><g transform="translate(2000,0)"><use xlink:href="#E273-MJMAIN-32"></use><use xlink:href="#E273-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E273-MJMAIN-32" x="778" y="0"></use><use xlink:href="#E273-MJMAIN-2E" x="1278" y="0"></use><use xlink:href="#E273-MJMAIN-35" x="1556" y="0"></use><g transform="translate(2056,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(2886,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">更</text></g><g transform="translate(3717,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">新</text></g><g transform="translate(4547,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">资</text></g><g transform="translate(5378,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">格</text></g><g transform="translate(6209,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">迹</text></g><g transform="translate(7039,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(7870,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">状</text></g><g transform="translate(8701,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">态</text></g><g transform="translate(9531,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">价</text></g><g transform="translate(10362,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">值</text></g><g transform="translate(11193,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">评</text></g><g transform="translate(12023,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">估</text></g><g transform="translate(12854,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">：</text></g><g transform="translate(13685,0)"><use xlink:href="#E273-MJMAINB-7A" x="0" y="0"></use><use xlink:href="#E273-MJMAIN-2190" x="788" y="0"></use><use xlink:href="#E273-MJMATHI-3B3" x="2066" y="0"></use><use xlink:href="#E273-MJMATHI-3BB" x="2609" y="0"></use><use xlink:href="#E273-MJMAINB-7A" x="3192" y="0"></use><use xlink:href="#E273-MJMAIN-2B" x="3925" y="0"></use><use xlink:href="#E273-MJMAIN-2207" x="4926" y="0"></use><use xlink:href="#E273-MJMATHI-76" x="5759" y="0"></use><use xlink:href="#E273-MJMAIN-28" x="6244" y="0"></use><use xlink:href="#E273-MJMATHI-53" x="6633" y="0"></use><use xlink:href="#E273-MJMAIN-3B" x="7278" y="0"></use><use xlink:href="#E273-MJMAINB-77" x="7722" y="0"></use><use xlink:href="#E273-MJMAIN-29" x="8553" y="0"></use></g><g transform="translate(22627,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">，</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">期</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">望</text></g><use xlink:href="#E273-MJMAIN-53" x="2991" y="0"></use><use xlink:href="#E273-MJMAIN-41" x="3547" y="0"></use><use xlink:href="#E273-MJMAIN-52" x="4297" y="0"></use><use xlink:href="#E273-MJMAIN-53" x="5033" y="0"></use><use xlink:href="#E273-MJMAIN-41" x="5589" y="0"></use><g transform="translate(6589,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">算</text></g><g transform="translate(7420,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">法</text></g></g></g></g><g transform="translate(0,-496)"><g transform="translate(4444,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">和</text><use xlink:href="#E273-MJMAIN-51" x="1080" y="0"></use><use xlink:href="#E273-MJMAIN-28" x="1858" y="0"></use><use xlink:href="#E273-MJMATHI-3BB" x="2247" y="0"></use><g transform="translate(2830,0)"><use xlink:href="#E273-MJMAIN-29"></use><g transform="translate(639,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">学</text></g><g transform="translate(1469,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">习</text></g><g transform="translate(2300,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">：</text></g></g><g transform="translate(5961,0)"><use xlink:href="#E273-MJMAINB-7A" x="0" y="0"></use><use xlink:href="#E273-MJMAIN-2190" x="788" y="0"></use><use xlink:href="#E273-MJMATHI-3B3" x="2066" y="0"></use><use xlink:href="#E273-MJMATHI-3BB" x="2609" y="0"></use><use xlink:href="#E273-MJMAINB-7A" x="3192" y="0"></use><use xlink:href="#E273-MJMAIN-2B" x="3925" y="0"></use><use xlink:href="#E273-MJMAIN-2207" x="4926" y="0"></use><use xlink:href="#E273-MJMATHI-71" x="5759" y="0"></use><use xlink:href="#E273-MJMAIN-28" x="6219" y="0"></use><use xlink:href="#E273-MJMATHI-53" x="6608" y="0"></use><use xlink:href="#E273-MJMAIN-2C" x="7253" y="0"></use><use xlink:href="#E273-MJMATHI-41" x="7697" y="0"></use><use xlink:href="#E273-MJMAIN-3B" x="8447" y="0"></use><use xlink:href="#E273-MJMAINB-77" x="8892" y="0"></use><use xlink:href="#E273-MJMAIN-29" x="9723" y="0"></use></g><g transform="translate(16073,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">；</text></g></g></g></g><g transform="translate(0,-1846)"><g transform="translate(2000,0)"><use xlink:href="#E273-MJMAIN-32"></use><use xlink:href="#E273-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E273-MJMAIN-32" x="778" y="0"></use><use xlink:href="#E273-MJMAIN-2E" x="1278" y="0"></use><use xlink:href="#E273-MJMAIN-35" x="1556" y="0"></use><g transform="translate(2056,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(2886,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">更</text></g><g transform="translate(3717,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">新</text></g><g transform="translate(4547,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">价</text></g><g transform="translate(5378,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">值</text></g><g transform="translate(6209,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(7039,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">状</text></g><g transform="translate(7870,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">态</text></g><g transform="translate(8701,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">价</text></g><g transform="translate(9531,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">值</text></g><g transform="translate(10362,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">评</text></g><g transform="translate(11193,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">估</text></g><g transform="translate(12023,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">：</text></g><g transform="translate(12854,0)"><use xlink:href="#E273-MJMAINB-77" x="0" y="0"></use><use xlink:href="#E273-MJMAIN-2190" x="1108" y="0"></use><use xlink:href="#E273-MJMAINB-77" x="2386" y="0"></use><use xlink:href="#E273-MJMAIN-2B" x="3439" y="0"></use><use xlink:href="#E273-MJMATHI-3B1" x="4440" y="0"></use><use xlink:href="#E273-MJMAIN-5B" x="5080" y="0"></use><use xlink:href="#E273-MJMATHI-55" x="5358" y="0"></use><use xlink:href="#E273-MJMAIN-2212" x="6347" y="0"></use><use xlink:href="#E273-MJMATHI-76" x="7347" y="0"></use><use xlink:href="#E273-MJMAIN-28" x="7832" y="0"></use><use xlink:href="#E273-MJMATHI-53" x="8221" y="0"></use><use xlink:href="#E273-MJMAIN-3B" x="8866" y="0"></use><use xlink:href="#E273-MJMAINB-77" x="9311" y="0"></use><use xlink:href="#E273-MJMAIN-29" x="10142" y="0"></use><use xlink:href="#E273-MJMAIN-5D" x="10531" y="0"></use><use xlink:href="#E273-MJMAINB-7A" x="10809" y="0"></use></g><g transform="translate(24174,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">，</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">期</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">望</text></g><use xlink:href="#E273-MJMAIN-53" x="2991" y="0"></use><use xlink:href="#E273-MJMAIN-41" x="3547" y="0"></use><use xlink:href="#E273-MJMAIN-52" x="4297" y="0"></use><use xlink:href="#E273-MJMAIN-53" x="5033" y="0"></use><use xlink:href="#E273-MJMAIN-41" x="5589" y="0"></use><g transform="translate(6589,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">算</text></g><g transform="translate(7420,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">法</text></g></g></g></g><g transform="translate(0,-3196)"><g transform="translate(4444,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">和</text><use xlink:href="#E273-MJMAIN-51" x="1080" y="0"></use><use xlink:href="#E273-MJMAIN-28" x="1858" y="0"></use><use xlink:href="#E273-MJMATHI-3BB" x="2247" y="0"></use><g transform="translate(2830,0)"><use xlink:href="#E273-MJMAIN-29"></use><g transform="translate(639,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">学</text></g><g transform="translate(1469,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">习</text></g><g transform="translate(2300,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">：</text></g></g><g transform="translate(5961,0)"><use xlink:href="#E273-MJMAINB-77" x="0" y="0"></use><use xlink:href="#E273-MJMAIN-2190" x="1108" y="0"></use><use xlink:href="#E273-MJMAINB-77" x="2386" y="0"></use><use xlink:href="#E273-MJMAIN-2B" x="3439" y="0"></use><use xlink:href="#E273-MJMATHI-3B1" x="4440" y="0"></use><use xlink:href="#E273-MJMAIN-5B" x="5080" y="0"></use><use xlink:href="#E273-MJMATHI-55" x="5358" y="0"></use><use xlink:href="#E273-MJMAIN-2212" x="6347" y="0"></use><use xlink:href="#E273-MJMATHI-71" x="7347" y="0"></use><use xlink:href="#E273-MJMAIN-28" x="7807" y="0"></use><use xlink:href="#E273-MJMATHI-53" x="8196" y="0"></use><use xlink:href="#E273-MJMAIN-2C" x="8841" y="0"></use><use xlink:href="#E273-MJMATHI-41" x="9286" y="0"></use><use xlink:href="#E273-MJMAIN-3B" x="10036" y="0"></use><use xlink:href="#E273-MJMAINB-77" x="10480" y="0"></use><use xlink:href="#E273-MJMAIN-29" x="11311" y="0"></use><use xlink:href="#E273-MJMAIN-5D" x="11700" y="0"></use><use xlink:href="#E273-MJMAINB-7A" x="11978" y="0"></use></g><g transform="translate(18451,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">；</text></g></g></g></g><g transform="translate(0,-4555)"><g transform="translate(2000,0)"><use xlink:href="#E273-MJMAIN-32"></use><use xlink:href="#E273-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E273-MJMAIN-32" x="778" y="0"></use><use xlink:href="#E273-MJMAIN-2E" x="1278" y="0"></use><use xlink:href="#E273-MJMAIN-36" x="1556" y="0"></use><g transform="translate(2306,0)"><use xlink:href="#E273-MJMATHI-53" x="444" y="0"></use><use xlink:href="#E273-MJMAIN-2190" x="1367" y="0"></use><g transform="translate(2645,0)"><use xlink:href="#E273-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E273-MJMAIN-2032" x="925" y="583"></use></g></g><g transform="translate(5900,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">。</text></g></g></g></g><g transform="translate(0,-5855)"><g><rect fill="black" stroke="none" width="41597" height="100" x="0" y="-500"></rect></g></g></g></g></g></svg></span></div><script type="math/tex; mode=display" id="MathJax-Element-246">\; \\ \; \\
\large \textbf{算法 6-6   TD($\lambda$) 估计状态价值或期望 SARSA($\lambda$) 算法或 Q($\lambda$) 学习} \\
\begin{split}
\rule[5pt]{10mm}{0.1em} &\rule[5pt]{265mm}{0.1em} \\
&\cdots \quad \text{同算法 6-4} \quad \cdots \\
&\text{2.1（初始化）$\bold z \leftarrow \bold 0$ ，选择状态 $S$ 。} \\
&\cdots \quad \text{同算法 6-4} \quad \cdots \\
&\qquad \text{2.2.5（更新资格迹）状态价值评估：$\bold z \leftarrow \gamma\lambda\bold z + \nabla v(S;\bold w)$ ，期望 SARSA 算法} \\
&\qquad \qquad \;\, \text{和 Q($\lambda$) 学习：$\bold z \leftarrow \gamma\lambda\bold z + \nabla q(S,A;\bold w)$ ；}\\
&\qquad \text{2.2.5（更新价值）状态价值评估：$\bold w \leftarrow \bold w + \alpha[U-v(S;\bold w)]\bold z$ ，期望 SARSA 算法} \\
&\qquad \qquad \;\, \text{和 Q($\lambda$) 学习：$\bold w \leftarrow \bold w + \alpha[U-q(S,A;\bold w)]\bold z$ ；}\\
&\qquad \text{2.2.6 $\;\, S \leftarrow S'$ 。}\\
\rule[-5pt]{10mm}{0.1em} &\rule[-5pt]{265mm}{0.1em}
\end{split}
\; \\ \; \\</script></div></div><h3><a name="二线性近似" class="md-header-anchor"></a><span>二、线性近似</span></h3><p><strong><span>线性近似</span></strong><span>是用许多特征向量的线性组合来近似价值函数，特征向量则依赖于输入（即状态或动作状态对），以动作价值近似为例，可以为每个状态动作对定义多个不同的特征 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="26.548ex" height="2.807ex" viewBox="0 -832.7 11430.3 1208.4" role="img" focusable="false" style="vertical-align: -0.873ex;"><defs><path stroke-width="0" id="E310-MJMAINB-78" d="M227 0Q212 3 121 3Q40 3 28 0H21V62H117L245 213L109 382H26V444H34Q49 441 143 441Q247 441 265 444H274V382H246L281 339Q315 297 316 297Q320 297 354 341L389 382H352V444H360Q375 441 466 441Q547 441 559 444H566V382H471L355 246L504 63L545 62H586V0H578Q563 3 469 3Q365 3 347 0H338V62H366Q366 63 326 112T285 163L198 63L217 62H235V0H227Z"></path><path stroke-width="0" id="E310-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E310-MJMATHI-73" d="M131 289Q131 321 147 354T203 415T300 442Q362 442 390 415T419 355Q419 323 402 308T364 292Q351 292 340 300T328 326Q328 342 337 354T354 372T367 378Q368 378 368 379Q368 382 361 388T336 399T297 405Q249 405 227 379T204 326Q204 301 223 291T278 274T330 259Q396 230 396 163Q396 135 385 107T352 51T289 7T195 -10Q118 -10 86 19T53 87Q53 126 74 143T118 160Q133 160 146 151T160 120Q160 94 142 76T111 58Q109 57 108 57T107 55Q108 52 115 47T146 34T201 27Q237 27 263 38T301 66T318 97T323 122Q323 150 302 164T254 181T195 196T148 231Q131 256 131 289Z"></path><path stroke-width="0" id="E310-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E310-MJMATHI-61" d="M33 157Q33 258 109 349T280 441Q331 441 370 392Q386 422 416 422Q429 422 439 414T449 394Q449 381 412 234T374 68Q374 43 381 35T402 26Q411 27 422 35Q443 55 463 131Q469 151 473 152Q475 153 483 153H487Q506 153 506 144Q506 138 501 117T481 63T449 13Q436 0 417 -8Q409 -10 393 -10Q359 -10 336 5T306 36L300 51Q299 52 296 50Q294 48 292 46Q233 -10 172 -10Q117 -10 75 30T33 157ZM351 328Q351 334 346 350T323 385T277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q217 26 254 59T298 110Q300 114 325 217T351 328Z"></path><path stroke-width="0" id="E310-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E310-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E310-MJMATHI-78" d="M52 289Q59 331 106 386T222 442Q257 442 286 424T329 379Q371 442 430 442Q467 442 494 420T522 361Q522 332 508 314T481 292T458 288Q439 288 427 299T415 328Q415 374 465 391Q454 404 425 404Q412 404 406 402Q368 386 350 336Q290 115 290 78Q290 50 306 38T341 26Q378 26 414 59T463 140Q466 150 469 151T485 153H489Q504 153 504 145Q504 144 502 134Q486 77 440 33T333 -11Q263 -11 227 52Q186 -10 133 -10H127Q78 -10 57 16T35 71Q35 103 54 123T99 143Q142 143 142 101Q142 81 130 66T107 46T94 41L91 40Q91 39 97 36T113 29T132 26Q168 26 194 71Q203 87 217 139T245 247T261 313Q266 340 266 352Q266 380 251 392T217 404Q177 404 142 372T93 290Q91 281 88 280T72 278H58Q52 284 52 289Z"></path><path stroke-width="0" id="E310-MJMATHI-6A" d="M297 596Q297 627 318 644T361 661Q378 661 389 651T403 623Q403 595 384 576T340 557Q322 557 310 567T297 596ZM288 376Q288 405 262 405Q240 405 220 393T185 362T161 325T144 293L137 279Q135 278 121 278H107Q101 284 101 286T105 299Q126 348 164 391T252 441Q253 441 260 441T272 442Q296 441 316 432Q341 418 354 401T367 348V332L318 133Q267 -67 264 -75Q246 -125 194 -164T75 -204Q25 -204 7 -183T-12 -137Q-12 -110 7 -91T53 -71Q70 -71 82 -81T95 -112Q95 -148 63 -167Q69 -168 77 -168Q111 -168 139 -140T182 -74L193 -32Q204 11 219 72T251 197T278 308T289 365Q289 372 288 376Z"></path><path stroke-width="0" id="E310-MJMAIN-3A" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 84 95 102T138 120Q162 120 180 104T199 61Q199 36 182 18T139 0T96 17T78 60Z"></path><path stroke-width="0" id="E310-MJMAIN-2208" d="M84 250Q84 372 166 450T360 539Q361 539 377 539T419 540T469 540H568Q583 532 583 520Q583 511 570 501L466 500Q355 499 329 494Q280 482 242 458T183 409T147 354T129 306T124 272V270H568Q583 262 583 250T568 230H124V228Q124 207 134 177T167 112T231 48T328 7Q355 1 466 0H570Q583 -10 583 -20Q583 -32 568 -40H471Q464 -40 446 -40T417 -41Q262 -41 172 45Q84 127 84 250Z"></path><path stroke-width="0" id="E310-MJCAL-4A" d="M148 78Q148 16 189 -17T286 -50Q319 -50 348 -33T396 10T426 59T444 101L471 204Q498 306 521 372Q575 532 649 605L659 614H591Q517 613 494 607Q433 591 400 550T360 477Q353 454 325 437T275 419Q256 419 260 435Q280 523 376 597T583 681Q603 683 713 683H830Q839 674 839 671Q839 654 810 634T754 614Q735 614 721 601Q688 571 654 495T600 351T561 209T541 132Q507 29 412 -45T213 -119Q141 -119 94 -77T47 33Q47 55 50 69T58 90T71 103Q105 131 135 131Q152 131 152 120Q152 119 151 114T149 99T148 78Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E310-MJMAINB-78" x="0" y="0"></use><use xlink:href="#E310-MJMAIN-28" x="607" y="0"></use><use xlink:href="#E310-MJMATHI-73" x="996" y="0"></use><use xlink:href="#E310-MJMAIN-2C" x="1465" y="0"></use><use xlink:href="#E310-MJMATHI-61" x="1909" y="0"></use><use xlink:href="#E310-MJMAIN-29" x="2438" y="0"></use><use xlink:href="#E310-MJMAIN-3D" x="3105" y="0"></use><use xlink:href="#E310-MJMAIN-28" x="4161" y="0"></use><g transform="translate(4550,0)"><use xlink:href="#E310-MJMATHI-78" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E310-MJMATHI-6A" x="808" y="-213"></use></g><use xlink:href="#E310-MJMAIN-28" x="5513" y="0"></use><use xlink:href="#E310-MJMATHI-73" x="5902" y="0"></use><use xlink:href="#E310-MJMAIN-2C" x="6371" y="0"></use><use xlink:href="#E310-MJMATHI-61" x="6816" y="0"></use><use xlink:href="#E310-MJMAIN-29" x="7345" y="0"></use><use xlink:href="#E310-MJMAIN-3A" x="8011" y="0"></use><use xlink:href="#E310-MJMATHI-6A" x="8567" y="0"></use><use xlink:href="#E310-MJMAIN-2208" x="9257" y="0"></use><use xlink:href="#E310-MJCAL-4A" x="10202" y="0"></use><use xlink:href="#E310-MJMAIN-29" x="11041" y="0"></use></g></svg></span><script type="math/tex">\bold x(s,a)=(x_j(s,a):j \in \mathcal J)</script><span> ，进而定义近似函数为这些特征的线性组合，即：</span></p><div contenteditable="false" spellcheck="false" class="mathjax-block md-end-block md-math-block md-rawblock" id="mathjax-n27" cid="n27" mdtype="math_block"><div class="md-rawblock-container md-math-container" tabindex="-1"><div class="MathJax_SVG_Display"><span class="MathJax_SVG" id="MathJax-Element-247-Frame" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="98.296ex" height="5.604ex" viewBox="0 -1455.6 42321.7 2412.9" role="img" focusable="false" style="vertical-align: -2.223ex; max-width: 100%;"><defs><path stroke-width="0" id="E274-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E274-MJMAIN-33" d="M127 463Q100 463 85 480T69 524Q69 579 117 622T233 665Q268 665 277 664Q351 652 390 611T430 522Q430 470 396 421T302 350L299 348Q299 347 308 345T337 336T375 315Q457 262 457 175Q457 96 395 37T238 -22Q158 -22 100 21T42 130Q42 158 60 175T105 193Q133 193 151 175T169 130Q169 119 166 110T159 94T148 82T136 74T126 70T118 67L114 66Q165 21 238 21Q293 21 321 74Q338 107 338 175V195Q338 290 274 322Q259 328 213 329L171 330L168 332Q166 335 166 348Q166 366 174 366Q202 366 232 371Q266 376 294 413T322 525V533Q322 590 287 612Q265 626 240 626Q208 626 181 615T143 592T132 580H135Q138 579 143 578T153 573T165 566T175 555T183 540T186 520Q186 498 172 481T127 463Z"></path><path stroke-width="0" id="E274-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E274-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E274-MJMATHI-73" d="M131 289Q131 321 147 354T203 415T300 442Q362 442 390 415T419 355Q419 323 402 308T364 292Q351 292 340 300T328 326Q328 342 337 354T354 372T367 378Q368 378 368 379Q368 382 361 388T336 399T297 405Q249 405 227 379T204 326Q204 301 223 291T278 274T330 259Q396 230 396 163Q396 135 385 107T352 51T289 7T195 -10Q118 -10 86 19T53 87Q53 126 74 143T118 160Q133 160 146 151T160 120Q160 94 142 76T111 58Q109 57 108 57T107 55Q108 52 115 47T146 34T201 27Q237 27 263 38T301 66T318 97T323 122Q323 150 302 164T254 181T195 196T148 231Q131 256 131 289Z"></path><path stroke-width="0" id="E274-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E274-MJMATHI-61" d="M33 157Q33 258 109 349T280 441Q331 441 370 392Q386 422 416 422Q429 422 439 414T449 394Q449 381 412 234T374 68Q374 43 381 35T402 26Q411 27 422 35Q443 55 463 131Q469 151 473 152Q475 153 483 153H487Q506 153 506 144Q506 138 501 117T481 63T449 13Q436 0 417 -8Q409 -10 393 -10Q359 -10 336 5T306 36L300 51Q299 52 296 50Q294 48 292 46Q233 -10 172 -10Q117 -10 75 30T33 157ZM351 328Q351 334 346 350T323 385T277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q217 26 254 59T298 110Q300 114 325 217T351 328Z"></path><path stroke-width="0" id="E274-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E274-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E274-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E274-MJMAIN-5B" d="M118 -250V750H255V710H158V-210H255V-250H118Z"></path><path stroke-width="0" id="E274-MJMAINB-78" d="M227 0Q212 3 121 3Q40 3 28 0H21V62H117L245 213L109 382H26V444H34Q49 441 143 441Q247 441 265 444H274V382H246L281 339Q315 297 316 297Q320 297 354 341L389 382H352V444H360Q375 441 466 441Q547 441 559 444H566V382H471L355 246L504 63L545 62H586V0H578Q563 3 469 3Q365 3 347 0H338V62H366Q366 63 326 112T285 163L198 63L217 62H235V0H227Z"></path><path stroke-width="0" id="E274-MJMAIN-5D" d="M22 710V750H159V-250H22V-210H119V710H22Z"></path><path stroke-width="0" id="E274-MJMATHI-54" d="M40 437Q21 437 21 445Q21 450 37 501T71 602L88 651Q93 669 101 677H569H659Q691 677 697 676T704 667Q704 661 687 553T668 444Q668 437 649 437Q640 437 637 437T631 442L629 445Q629 451 635 490T641 551Q641 586 628 604T573 629Q568 630 515 631Q469 631 457 630T439 622Q438 621 368 343T298 60Q298 48 386 46Q418 46 427 45T436 36Q436 31 433 22Q429 4 424 1L422 0Q419 0 415 0Q410 0 363 1T228 2Q99 2 64 0H49Q43 6 43 9T45 27Q49 40 55 46H83H94Q174 46 189 55Q190 56 191 56Q196 59 201 76T241 233Q258 301 269 344Q339 619 339 625Q339 630 310 630H279Q212 630 191 624Q146 614 121 583T67 467Q60 445 57 441T43 437H40Z"></path><path stroke-width="0" id="E274-MJSZ2-2211" d="M60 948Q63 950 665 950H1267L1325 815Q1384 677 1388 669H1348L1341 683Q1320 724 1285 761Q1235 809 1174 838T1033 881T882 898T699 902H574H543H251L259 891Q722 258 724 252Q725 250 724 246Q721 243 460 -56L196 -356Q196 -357 407 -357Q459 -357 548 -357T676 -358Q812 -358 896 -353T1063 -332T1204 -283T1307 -196Q1328 -170 1348 -124H1388Q1388 -125 1381 -145T1356 -210T1325 -294L1267 -449L666 -450Q64 -450 61 -448Q55 -446 55 -439Q55 -437 57 -433L590 177Q590 178 557 222T452 366T322 544L56 909L55 924Q55 945 60 948Z"></path><path stroke-width="0" id="E274-MJMATHI-6A" d="M297 596Q297 627 318 644T361 661Q378 661 389 651T403 623Q403 595 384 576T340 557Q322 557 310 567T297 596ZM288 376Q288 405 262 405Q240 405 220 393T185 362T161 325T144 293L137 279Q135 278 121 278H107Q101 284 101 286T105 299Q126 348 164 391T252 441Q253 441 260 441T272 442Q296 441 316 432Q341 418 354 401T367 348V332L318 133Q267 -67 264 -75Q246 -125 194 -164T75 -204Q25 -204 7 -183T-12 -137Q-12 -110 7 -91T53 -71Q70 -71 82 -81T95 -112Q95 -148 63 -167Q69 -168 77 -168Q111 -168 139 -140T182 -74L193 -32Q204 11 219 72T251 197T278 308T289 365Q289 372 288 376Z"></path><path stroke-width="0" id="E274-MJMAIN-2208" d="M84 250Q84 372 166 450T360 539Q361 539 377 539T419 540T469 540H568Q583 532 583 520Q583 511 570 501L466 500Q355 499 329 494Q280 482 242 458T183 409T147 354T129 306T124 272V270H568Q583 262 583 250T568 230H124V228Q124 207 134 177T167 112T231 48T328 7Q355 1 466 0H570Q583 -10 583 -20Q583 -32 568 -40H471Q464 -40 446 -40T417 -41Q262 -41 172 45Q84 127 84 250Z"></path><path stroke-width="0" id="E274-MJCAL-4A" d="M148 78Q148 16 189 -17T286 -50Q319 -50 348 -33T396 10T426 59T444 101L471 204Q498 306 521 372Q575 532 649 605L659 614H591Q517 613 494 607Q433 591 400 550T360 477Q353 454 325 437T275 419Q256 419 260 435Q280 523 376 597T583 681Q603 683 713 683H830Q839 674 839 671Q839 654 810 634T754 614Q735 614 721 601Q688 571 654 495T600 351T561 209T541 132Q507 29 412 -45T213 -119Q141 -119 94 -77T47 33Q47 55 50 69T58 90T71 103Q105 131 135 131Q152 131 152 120Q152 119 151 114T149 99T148 78Z"></path><path stroke-width="0" id="E274-MJMATHI-78" d="M52 289Q59 331 106 386T222 442Q257 442 286 424T329 379Q371 442 430 442Q467 442 494 420T522 361Q522 332 508 314T481 292T458 288Q439 288 427 299T415 328Q415 374 465 391Q454 404 425 404Q412 404 406 402Q368 386 350 336Q290 115 290 78Q290 50 306 38T341 26Q378 26 414 59T463 140Q466 150 469 151T485 153H489Q504 153 504 145Q504 144 502 134Q486 77 440 33T333 -11Q263 -11 227 52Q186 -10 133 -10H127Q78 -10 57 16T35 71Q35 103 54 123T99 143Q142 143 142 101Q142 81 130 66T107 46T94 41L91 40Q91 39 97 36T113 29T132 26Q168 26 194 71Q203 87 217 139T245 247T261 313Q266 340 266 352Q266 380 251 392T217 404Q177 404 142 372T93 290Q91 281 88 280T72 278H58Q52 284 52 289Z"></path><path stroke-width="0" id="E274-MJMATHI-77" d="M580 385Q580 406 599 424T641 443Q659 443 674 425T690 368Q690 339 671 253Q656 197 644 161T609 80T554 12T482 -11Q438 -11 404 5T355 48Q354 47 352 44Q311 -11 252 -11Q226 -11 202 -5T155 14T118 53T104 116Q104 170 138 262T173 379Q173 380 173 381Q173 390 173 393T169 400T158 404H154Q131 404 112 385T82 344T65 302T57 280Q55 278 41 278H27Q21 284 21 287Q21 293 29 315T52 366T96 418T161 441Q204 441 227 416T250 358Q250 340 217 250T184 111Q184 65 205 46T258 26Q301 26 334 87L339 96V119Q339 122 339 128T340 136T341 143T342 152T345 165T348 182T354 206T362 238T373 281Q402 395 406 404Q419 431 449 431Q468 431 475 421T483 402Q483 389 454 274T422 142Q420 131 420 107V100Q420 85 423 71T442 42T487 26Q558 26 600 148Q609 171 620 213T632 273Q632 306 619 325T593 357T580 385Z"></path><path stroke-width="0" id="E274-MJCAL-53" d="M554 512Q536 512 536 522Q536 525 539 539T542 564Q542 588 528 604Q515 616 482 625T410 635Q374 635 349 624T312 594T295 561T290 532Q290 505 303 482T342 442T378 419T409 404Q435 391 451 383T494 357T535 323T562 282T574 231Q574 133 464 56T220 -22Q138 -22 78 21T18 123Q18 184 61 227T156 274Q178 274 178 263Q178 260 177 258Q172 247 164 239T151 227T136 218L127 213L124 202Q118 186 118 163Q120 124 165 86T292 48Q374 48 423 86T473 186V193Q473 267 347 327Q268 364 239 389Q191 431 191 486Q191 547 242 600T356 679T470 705Q472 705 478 705T489 704Q551 704 596 682T642 610Q642 566 621 545Q592 516 554 512Z"></path><path stroke-width="0" id="E274-MJCAL-41" d="M576 668Q576 688 606 708T660 728Q676 728 675 712V571Q675 409 688 252Q696 122 720 57Q722 53 723 50T728 46T732 43T737 41T743 39L754 45Q788 61 803 61Q819 61 819 47Q818 43 814 35Q799 15 755 -7T675 -30Q659 -30 648 -25T630 -8T621 11T614 34Q603 77 599 106T594 146T591 160V163H460L329 164L316 145Q241 35 196 -7T119 -50T59 -24T30 43Q30 75 46 100T74 125Q81 125 83 120T88 104T96 84Q118 57 151 57Q189 57 277 182Q432 400 542 625L559 659H567Q574 659 575 660T576 668ZM584 249Q579 333 577 386T575 473T574 520V581L563 560Q497 426 412 290L372 228L370 224H371L383 228L393 232H586L584 249Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><g transform="translate(41043,0)"><g id="mjx-eqn-3" transform="translate(0,446)"><use xlink:href="#E274-MJMAIN-28"></use><use xlink:href="#E274-MJMAIN-33" x="389" y="0"></use><use xlink:href="#E274-MJMAIN-29" x="889" y="0"></use></g></g><g transform="translate(8548,0)"><g transform="translate(-19,0)"><g transform="translate(0,446)"><use xlink:href="#E274-MJMATHI-71" x="0" y="0"></use><use xlink:href="#E274-MJMAIN-28" x="460" y="0"></use><use xlink:href="#E274-MJMATHI-73" x="849" y="0"></use><use xlink:href="#E274-MJMAIN-2C" x="1318" y="0"></use><use xlink:href="#E274-MJMATHI-61" x="1762" y="0"></use><use xlink:href="#E274-MJMAIN-3B" x="2291" y="0"></use><use xlink:href="#E274-MJMAINB-77" x="2736" y="0"></use><use xlink:href="#E274-MJMAIN-29" x="3567" y="0"></use><use xlink:href="#E274-MJMAIN-3D" x="4234" y="0"></use><use xlink:href="#E274-MJMAIN-5B" x="5289" y="0"></use><use xlink:href="#E274-MJMAINB-78" x="5567" y="0"></use><use xlink:href="#E274-MJMAIN-28" x="6174" y="0"></use><use xlink:href="#E274-MJMATHI-73" x="6563" y="0"></use><use xlink:href="#E274-MJMAIN-2C" x="7032" y="0"></use><use xlink:href="#E274-MJMATHI-61" x="7477" y="0"></use><use xlink:href="#E274-MJMAIN-29" x="8006" y="0"></use><g transform="translate(8395,0)"><use xlink:href="#E274-MJMAIN-5D" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E274-MJMATHI-54" x="393" y="583"></use></g><use xlink:href="#E274-MJMAINB-77" x="9271" y="0"></use><use xlink:href="#E274-MJMAIN-3D" x="10380" y="0"></use><g transform="translate(11435,0)"><use xlink:href="#E274-MJSZ2-2211" x="0" y="0"></use><g transform="translate(43,-1100)"><use transform="scale(0.707)" xlink:href="#E274-MJMATHI-6A" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E274-MJMAIN-2208" x="412" y="0"></use><use transform="scale(0.707)" xlink:href="#E274-MJCAL-4A" x="1079" y="0"></use></g></g><g transform="translate(13046,0)"><use xlink:href="#E274-MJMATHI-78" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E274-MJMATHI-6A" x="808" y="-213"></use></g><use xlink:href="#E274-MJMAIN-28" x="14009" y="0"></use><use xlink:href="#E274-MJMATHI-73" x="14398" y="0"></use><use xlink:href="#E274-MJMAIN-2C" x="14867" y="0"></use><use xlink:href="#E274-MJMATHI-61" x="15312" y="0"></use><use xlink:href="#E274-MJMAIN-29" x="15841" y="0"></use><g transform="translate(16230,0)"><use xlink:href="#E274-MJMATHI-77" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E274-MJMATHI-6A" x="1012" y="-213"></use></g><use xlink:href="#E274-MJMAIN-2C" x="17782" y="0"></use><use xlink:href="#E274-MJMATHI-73" x="20227" y="0"></use><use xlink:href="#E274-MJMAIN-2208" x="20973" y="0"></use><use xlink:href="#E274-MJCAL-53" x="21918" y="0"></use><use xlink:href="#E274-MJMAIN-2C" x="22560" y="0"></use><use xlink:href="#E274-MJMATHI-61" x="23005" y="0"></use><use xlink:href="#E274-MJMAIN-2208" x="23812" y="0"></use><use xlink:href="#E274-MJCAL-41" x="24756" y="0"></use></g></g></g></g></svg></span></div><script type="math/tex; mode=display" id="MathJax-Element-247">q(s,a;\bold w)=[\bold x(s,a)]^T\bold w = \sum_{j \in \mathcal J} x_j(s,a)w_j \;\,, \qquad s \in \mathcal S,a \in \mathcal A</script></div></div><p><span>对于状态函数也有类似的近似方法：</span></p><div contenteditable="false" spellcheck="false" class="mathjax-block md-end-block md-math-block md-rawblock" id="mathjax-n29" cid="n29" mdtype="math_block"><div class="md-rawblock-container md-math-container" tabindex="-1"><div class="MathJax_SVG_Display"><span class="MathJax_SVG" id="MathJax-Element-248-Frame" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="98.296ex" height="5.604ex" viewBox="0 -1455.6 42321.7 2412.9" role="img" focusable="false" style="vertical-align: -2.223ex; max-width: 100%;"><defs><path stroke-width="0" id="E275-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E275-MJMAIN-34" d="M462 0Q444 3 333 3Q217 3 199 0H190V46H221Q241 46 248 46T265 48T279 53T286 61Q287 63 287 115V165H28V211L179 442Q332 674 334 675Q336 677 355 677H373L379 671V211H471V165H379V114Q379 73 379 66T385 54Q393 47 442 46H471V0H462ZM293 211V545L74 212L183 211H293Z"></path><path stroke-width="0" id="E275-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E275-MJMATHI-76" d="M173 380Q173 405 154 405Q130 405 104 376T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Q21 294 29 316T53 368T97 419T160 441Q202 441 225 417T249 361Q249 344 246 335Q246 329 231 291T200 202T182 113Q182 86 187 69Q200 26 250 26Q287 26 319 60T369 139T398 222T409 277Q409 300 401 317T383 343T365 361T357 383Q357 405 376 424T417 443Q436 443 451 425T467 367Q467 340 455 284T418 159T347 40T241 -11Q177 -11 139 22Q102 54 102 117Q102 148 110 181T151 298Q173 362 173 380Z"></path><path stroke-width="0" id="E275-MJMATHI-73" d="M131 289Q131 321 147 354T203 415T300 442Q362 442 390 415T419 355Q419 323 402 308T364 292Q351 292 340 300T328 326Q328 342 337 354T354 372T367 378Q368 378 368 379Q368 382 361 388T336 399T297 405Q249 405 227 379T204 326Q204 301 223 291T278 274T330 259Q396 230 396 163Q396 135 385 107T352 51T289 7T195 -10Q118 -10 86 19T53 87Q53 126 74 143T118 160Q133 160 146 151T160 120Q160 94 142 76T111 58Q109 57 108 57T107 55Q108 52 115 47T146 34T201 27Q237 27 263 38T301 66T318 97T323 122Q323 150 302 164T254 181T195 196T148 231Q131 256 131 289Z"></path><path stroke-width="0" id="E275-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E275-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E275-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E275-MJMAIN-5B" d="M118 -250V750H255V710H158V-210H255V-250H118Z"></path><path stroke-width="0" id="E275-MJMAINB-78" d="M227 0Q212 3 121 3Q40 3 28 0H21V62H117L245 213L109 382H26V444H34Q49 441 143 441Q247 441 265 444H274V382H246L281 339Q315 297 316 297Q320 297 354 341L389 382H352V444H360Q375 441 466 441Q547 441 559 444H566V382H471L355 246L504 63L545 62H586V0H578Q563 3 469 3Q365 3 347 0H338V62H366Q366 63 326 112T285 163L198 63L217 62H235V0H227Z"></path><path stroke-width="0" id="E275-MJMAIN-5D" d="M22 710V750H159V-250H22V-210H119V710H22Z"></path><path stroke-width="0" id="E275-MJMATHI-54" d="M40 437Q21 437 21 445Q21 450 37 501T71 602L88 651Q93 669 101 677H569H659Q691 677 697 676T704 667Q704 661 687 553T668 444Q668 437 649 437Q640 437 637 437T631 442L629 445Q629 451 635 490T641 551Q641 586 628 604T573 629Q568 630 515 631Q469 631 457 630T439 622Q438 621 368 343T298 60Q298 48 386 46Q418 46 427 45T436 36Q436 31 433 22Q429 4 424 1L422 0Q419 0 415 0Q410 0 363 1T228 2Q99 2 64 0H49Q43 6 43 9T45 27Q49 40 55 46H83H94Q174 46 189 55Q190 56 191 56Q196 59 201 76T241 233Q258 301 269 344Q339 619 339 625Q339 630 310 630H279Q212 630 191 624Q146 614 121 583T67 467Q60 445 57 441T43 437H40Z"></path><path stroke-width="0" id="E275-MJSZ2-2211" d="M60 948Q63 950 665 950H1267L1325 815Q1384 677 1388 669H1348L1341 683Q1320 724 1285 761Q1235 809 1174 838T1033 881T882 898T699 902H574H543H251L259 891Q722 258 724 252Q725 250 724 246Q721 243 460 -56L196 -356Q196 -357 407 -357Q459 -357 548 -357T676 -358Q812 -358 896 -353T1063 -332T1204 -283T1307 -196Q1328 -170 1348 -124H1388Q1388 -125 1381 -145T1356 -210T1325 -294L1267 -449L666 -450Q64 -450 61 -448Q55 -446 55 -439Q55 -437 57 -433L590 177Q590 178 557 222T452 366T322 544L56 909L55 924Q55 945 60 948Z"></path><path stroke-width="0" id="E275-MJMATHI-6A" d="M297 596Q297 627 318 644T361 661Q378 661 389 651T403 623Q403 595 384 576T340 557Q322 557 310 567T297 596ZM288 376Q288 405 262 405Q240 405 220 393T185 362T161 325T144 293L137 279Q135 278 121 278H107Q101 284 101 286T105 299Q126 348 164 391T252 441Q253 441 260 441T272 442Q296 441 316 432Q341 418 354 401T367 348V332L318 133Q267 -67 264 -75Q246 -125 194 -164T75 -204Q25 -204 7 -183T-12 -137Q-12 -110 7 -91T53 -71Q70 -71 82 -81T95 -112Q95 -148 63 -167Q69 -168 77 -168Q111 -168 139 -140T182 -74L193 -32Q204 11 219 72T251 197T278 308T289 365Q289 372 288 376Z"></path><path stroke-width="0" id="E275-MJMAIN-2208" d="M84 250Q84 372 166 450T360 539Q361 539 377 539T419 540T469 540H568Q583 532 583 520Q583 511 570 501L466 500Q355 499 329 494Q280 482 242 458T183 409T147 354T129 306T124 272V270H568Q583 262 583 250T568 230H124V228Q124 207 134 177T167 112T231 48T328 7Q355 1 466 0H570Q583 -10 583 -20Q583 -32 568 -40H471Q464 -40 446 -40T417 -41Q262 -41 172 45Q84 127 84 250Z"></path><path stroke-width="0" id="E275-MJCAL-4A" d="M148 78Q148 16 189 -17T286 -50Q319 -50 348 -33T396 10T426 59T444 101L471 204Q498 306 521 372Q575 532 649 605L659 614H591Q517 613 494 607Q433 591 400 550T360 477Q353 454 325 437T275 419Q256 419 260 435Q280 523 376 597T583 681Q603 683 713 683H830Q839 674 839 671Q839 654 810 634T754 614Q735 614 721 601Q688 571 654 495T600 351T561 209T541 132Q507 29 412 -45T213 -119Q141 -119 94 -77T47 33Q47 55 50 69T58 90T71 103Q105 131 135 131Q152 131 152 120Q152 119 151 114T149 99T148 78Z"></path><path stroke-width="0" id="E275-MJMATHI-78" d="M52 289Q59 331 106 386T222 442Q257 442 286 424T329 379Q371 442 430 442Q467 442 494 420T522 361Q522 332 508 314T481 292T458 288Q439 288 427 299T415 328Q415 374 465 391Q454 404 425 404Q412 404 406 402Q368 386 350 336Q290 115 290 78Q290 50 306 38T341 26Q378 26 414 59T463 140Q466 150 469 151T485 153H489Q504 153 504 145Q504 144 502 134Q486 77 440 33T333 -11Q263 -11 227 52Q186 -10 133 -10H127Q78 -10 57 16T35 71Q35 103 54 123T99 143Q142 143 142 101Q142 81 130 66T107 46T94 41L91 40Q91 39 97 36T113 29T132 26Q168 26 194 71Q203 87 217 139T245 247T261 313Q266 340 266 352Q266 380 251 392T217 404Q177 404 142 372T93 290Q91 281 88 280T72 278H58Q52 284 52 289Z"></path><path stroke-width="0" id="E275-MJMATHI-77" d="M580 385Q580 406 599 424T641 443Q659 443 674 425T690 368Q690 339 671 253Q656 197 644 161T609 80T554 12T482 -11Q438 -11 404 5T355 48Q354 47 352 44Q311 -11 252 -11Q226 -11 202 -5T155 14T118 53T104 116Q104 170 138 262T173 379Q173 380 173 381Q173 390 173 393T169 400T158 404H154Q131 404 112 385T82 344T65 302T57 280Q55 278 41 278H27Q21 284 21 287Q21 293 29 315T52 366T96 418T161 441Q204 441 227 416T250 358Q250 340 217 250T184 111Q184 65 205 46T258 26Q301 26 334 87L339 96V119Q339 122 339 128T340 136T341 143T342 152T345 165T348 182T354 206T362 238T373 281Q402 395 406 404Q419 431 449 431Q468 431 475 421T483 402Q483 389 454 274T422 142Q420 131 420 107V100Q420 85 423 71T442 42T487 26Q558 26 600 148Q609 171 620 213T632 273Q632 306 619 325T593 357T580 385Z"></path><path stroke-width="0" id="E275-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E275-MJCAL-53" d="M554 512Q536 512 536 522Q536 525 539 539T542 564Q542 588 528 604Q515 616 482 625T410 635Q374 635 349 624T312 594T295 561T290 532Q290 505 303 482T342 442T378 419T409 404Q435 391 451 383T494 357T535 323T562 282T574 231Q574 133 464 56T220 -22Q138 -22 78 21T18 123Q18 184 61 227T156 274Q178 274 178 263Q178 260 177 258Q172 247 164 239T151 227T136 218L127 213L124 202Q118 186 118 163Q120 124 165 86T292 48Q374 48 423 86T473 186V193Q473 267 347 327Q268 364 239 389Q191 431 191 486Q191 547 242 600T356 679T470 705Q472 705 478 705T489 704Q551 704 596 682T642 610Q642 566 621 545Q592 516 554 512Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><g transform="translate(41043,0)"><g id="mjx-eqn-4" transform="translate(0,446)"><use xlink:href="#E275-MJMAIN-28"></use><use xlink:href="#E275-MJMAIN-34" x="389" y="0"></use><use xlink:href="#E275-MJMAIN-29" x="889" y="0"></use></g></g><g transform="translate(11504,0)"><g transform="translate(-19,0)"><g transform="translate(0,446)"><use xlink:href="#E275-MJMATHI-76" x="0" y="0"></use><use xlink:href="#E275-MJMAIN-28" x="485" y="0"></use><use xlink:href="#E275-MJMATHI-73" x="874" y="0"></use><use xlink:href="#E275-MJMAIN-3B" x="1343" y="0"></use><use xlink:href="#E275-MJMAINB-77" x="1787" y="0"></use><use xlink:href="#E275-MJMAIN-29" x="2618" y="0"></use><use xlink:href="#E275-MJMAIN-3D" x="3285" y="0"></use><use xlink:href="#E275-MJMAIN-5B" x="4341" y="0"></use><use xlink:href="#E275-MJMAINB-78" x="4619" y="0"></use><use xlink:href="#E275-MJMAIN-28" x="5226" y="0"></use><use xlink:href="#E275-MJMATHI-73" x="5615" y="0"></use><use xlink:href="#E275-MJMAIN-29" x="6084" y="0"></use><g transform="translate(6473,0)"><use xlink:href="#E275-MJMAIN-5D" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E275-MJMATHI-54" x="393" y="583"></use></g><use xlink:href="#E275-MJMAINB-77" x="7349" y="0"></use><use xlink:href="#E275-MJMAIN-3D" x="8457" y="0"></use><g transform="translate(9513,0)"><use xlink:href="#E275-MJSZ2-2211" x="0" y="0"></use><g transform="translate(43,-1100)"><use transform="scale(0.707)" xlink:href="#E275-MJMATHI-6A" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E275-MJMAIN-2208" x="412" y="0"></use><use transform="scale(0.707)" xlink:href="#E275-MJCAL-4A" x="1079" y="0"></use></g></g><g transform="translate(11124,0)"><use xlink:href="#E275-MJMATHI-78" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E275-MJMATHI-6A" x="808" y="-213"></use></g><use xlink:href="#E275-MJMAIN-28" x="12087" y="0"></use><use xlink:href="#E275-MJMATHI-73" x="12476" y="0"></use><use xlink:href="#E275-MJMAIN-29" x="12945" y="0"></use><g transform="translate(13334,0)"><use xlink:href="#E275-MJMATHI-77" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E275-MJMATHI-6A" x="1012" y="-213"></use></g><use xlink:href="#E275-MJMAIN-2C" x="14886" y="0"></use><use xlink:href="#E275-MJMATHI-73" x="17331" y="0"></use><use xlink:href="#E275-MJMAIN-2208" x="18077" y="0"></use><use xlink:href="#E275-MJCAL-53" x="19022" y="0"></use></g></g></g></g></svg></span></div><script type="math/tex; mode=display" id="MathJax-Element-248">v(s;\bold w)=[\bold x(s)]^T\bold w = \sum_{j \in \mathcal J} x_j(s)w_j \;\,, \qquad s \in \mathcal S</script></div></div><p><span> 第三到五章的查表法可看做是线性近似的特例，例如对动作价值而言，可认为有 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="8.815ex" height="2.614ex" viewBox="0 -832.7 3795.4 1125.4" role="img" focusable="false" style="vertical-align: -0.68ex;"><defs><path stroke-width="0" id="E311-MJMAIN-7C" d="M139 -249H137Q125 -249 119 -235V251L120 737Q130 750 139 750Q152 750 159 735V-235Q151 -249 141 -249H139Z"></path><path stroke-width="0" id="E311-MJCAL-53" d="M554 512Q536 512 536 522Q536 525 539 539T542 564Q542 588 528 604Q515 616 482 625T410 635Q374 635 349 624T312 594T295 561T290 532Q290 505 303 482T342 442T378 419T409 404Q435 391 451 383T494 357T535 323T562 282T574 231Q574 133 464 56T220 -22Q138 -22 78 21T18 123Q18 184 61 227T156 274Q178 274 178 263Q178 260 177 258Q172 247 164 239T151 227T136 218L127 213L124 202Q118 186 118 163Q120 124 165 86T292 48Q374 48 423 86T473 186V193Q473 267 347 327Q268 364 239 389Q191 431 191 486Q191 547 242 600T356 679T470 705Q472 705 478 705T489 704Q551 704 596 682T642 610Q642 566 621 545Q592 516 554 512Z"></path><path stroke-width="0" id="E311-MJMAIN-D7" d="M630 29Q630 9 609 9Q604 9 587 25T493 118L389 222L284 117Q178 13 175 11Q171 9 168 9Q160 9 154 15T147 29Q147 36 161 51T255 146L359 250L255 354Q174 435 161 449T147 471Q147 480 153 485T168 490Q173 490 175 489Q178 487 284 383L389 278L493 382Q570 459 587 475T609 491Q630 491 630 471Q630 464 620 453T522 355L418 250L522 145Q606 61 618 48T630 29Z"></path><path stroke-width="0" id="E311-MJCAL-41" d="M576 668Q576 688 606 708T660 728Q676 728 675 712V571Q675 409 688 252Q696 122 720 57Q722 53 723 50T728 46T732 43T737 41T743 39L754 45Q788 61 803 61Q819 61 819 47Q818 43 814 35Q799 15 755 -7T675 -30Q659 -30 648 -25T630 -8T621 11T614 34Q603 77 599 106T594 146T591 160V163H460L329 164L316 145Q241 35 196 -7T119 -50T59 -24T30 43Q30 75 46 100T74 125Q81 125 83 120T88 104T96 84Q118 57 151 57Q189 57 277 182Q432 400 542 625L559 659H567Q574 659 575 660T576 668ZM584 249Q579 333 577 386T575 473T574 520V581L563 560Q497 426 412 290L372 228L370 224H371L383 228L393 232H586L584 249Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E311-MJMAIN-7C" x="0" y="0"></use><use xlink:href="#E311-MJCAL-53" x="278" y="0"></use><use xlink:href="#E311-MJMAIN-7C" x="920" y="0"></use><use xlink:href="#E311-MJMAIN-D7" x="1420" y="0"></use><use xlink:href="#E311-MJMAIN-7C" x="2420" y="0"></use><use xlink:href="#E311-MJCAL-41" x="2698" y="0"></use><use xlink:href="#E311-MJMAIN-7C" x="3517" y="0"></use></g></svg></span><script type="math/tex">|\mathcal S| \times |\mathcal A|</script><span> 个特征向量，每个向量形式为：</span></p><div contenteditable="false" spellcheck="false" class="mathjax-block md-end-block md-math-block md-rawblock" id="mathjax-n31" cid="n31" mdtype="math_block"><div class="md-rawblock-container md-math-container" tabindex="-1"><div class="MathJax_SVG_Display"><span class="MathJax_SVG" id="MathJax-Element-249-Frame" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="98.296ex" height="11.006ex" viewBox="0 -2618.6 42321.7 4738.7" role="img" focusable="false" style="vertical-align: -4.924ex; max-width: 100%;"><defs><path stroke-width="0" id="E276-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E276-MJMAIN-35" d="M164 157Q164 133 148 117T109 101H102Q148 22 224 22Q294 22 326 82Q345 115 345 210Q345 313 318 349Q292 382 260 382H254Q176 382 136 314Q132 307 129 306T114 304Q97 304 95 310Q93 314 93 485V614Q93 664 98 664Q100 666 102 666Q103 666 123 658T178 642T253 634Q324 634 389 662Q397 666 402 666Q410 666 410 648V635Q328 538 205 538Q174 538 149 544L139 546V374Q158 388 169 396T205 412T256 420Q337 420 393 355T449 201Q449 109 385 44T229 -22Q148 -22 99 32T50 154Q50 178 61 192T84 210T107 214Q132 214 148 197T164 157Z"></path><path stroke-width="0" id="E276-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E276-MJMAIN-30" d="M96 585Q152 666 249 666Q297 666 345 640T423 548Q460 465 460 320Q460 165 417 83Q397 41 362 16T301 -15T250 -22Q224 -22 198 -16T137 16T82 83Q39 165 39 320Q39 494 96 585ZM321 597Q291 629 250 629Q208 629 178 597Q153 571 145 525T137 333Q137 175 145 125T181 46Q209 16 250 16Q290 16 318 46Q347 76 354 130T362 333Q362 478 354 524T321 597Z"></path><path stroke-width="0" id="E276-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E276-MJMAIN-22EF" d="M78 250Q78 274 95 292T138 310Q162 310 180 294T199 251Q199 226 182 208T139 190T96 207T78 250ZM525 250Q525 274 542 292T585 310Q609 310 627 294T646 251Q646 226 629 208T586 190T543 207T525 250ZM972 250Q972 274 989 292T1032 310Q1056 310 1074 294T1093 251Q1093 226 1076 208T1033 190T990 207T972 250Z"></path><path stroke-width="0" id="E276-MJMAIN-31" d="M213 578L200 573Q186 568 160 563T102 556H83V602H102Q149 604 189 617T245 641T273 663Q275 666 285 666Q294 666 302 660V361L303 61Q310 54 315 52T339 48T401 46H427V0H416Q395 3 257 3Q121 3 100 0H88V46H114Q136 46 152 46T177 47T193 50T201 52T207 57T213 61V578Z"></path><path stroke-width="0" id="E276-MJMAIN-2191" d="M27 414Q17 414 17 433Q17 437 17 439T17 444T19 447T20 450T22 452T26 453T30 454T36 456Q80 467 120 494T180 549Q227 607 238 678Q240 694 251 694Q259 694 261 684Q261 677 265 659T284 608T320 549Q340 525 363 507T405 479T440 463T467 455T479 451Q483 447 483 433Q483 413 472 413Q467 413 458 416Q342 448 277 545L270 555V-179Q262 -193 252 -193H250H248Q236 -193 230 -179V555L223 545Q192 499 146 467T70 424T27 414Z"></path><path stroke-width="0" id="E276-MJMATHI-73" d="M131 289Q131 321 147 354T203 415T300 442Q362 442 390 415T419 355Q419 323 402 308T364 292Q351 292 340 300T328 326Q328 342 337 354T354 372T367 378Q368 378 368 379Q368 382 361 388T336 399T297 405Q249 405 227 379T204 326Q204 301 223 291T278 274T330 259Q396 230 396 163Q396 135 385 107T352 51T289 7T195 -10Q118 -10 86 19T53 87Q53 126 74 143T118 160Q133 160 146 151T160 120Q160 94 142 76T111 58Q109 57 108 57T107 55Q108 52 115 47T146 34T201 27Q237 27 263 38T301 66T318 97T323 122Q323 150 302 164T254 181T195 196T148 231Q131 256 131 289Z"></path><path stroke-width="0" id="E276-MJMATHI-61" d="M33 157Q33 258 109 349T280 441Q331 441 370 392Q386 422 416 422Q429 422 439 414T449 394Q449 381 412 234T374 68Q374 43 381 35T402 26Q411 27 422 35Q443 55 463 131Q469 151 473 152Q475 153 483 153H487Q506 153 506 144Q506 138 501 117T481 63T449 13Q436 0 417 -8Q409 -10 393 -10Q359 -10 336 5T306 36L300 51Q299 52 296 50Q294 48 292 46Q233 -10 172 -10Q117 -10 75 30T33 157ZM351 328Q351 334 346 350T323 385T277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q217 26 254 59T298 110Q300 114 325 217T351 328Z"></path><path stroke-width="0" id="E276-MJSZ4-239B" d="M837 1154Q843 1148 843 1145Q843 1141 818 1106T753 1002T667 841T574 604T494 299Q417 -84 417 -609Q417 -641 416 -647T411 -654Q409 -655 366 -655Q299 -655 297 -654Q292 -652 292 -643T291 -583Q293 -400 304 -242T347 110T432 470T574 813T785 1136Q787 1139 790 1142T794 1147T796 1150T799 1152T802 1153T807 1154T813 1154H819H837Z"></path><path stroke-width="0" id="E276-MJSZ4-239D" d="M843 -635Q843 -638 837 -644H820Q801 -644 800 -643Q792 -635 785 -626Q684 -503 605 -363T473 -75T385 216T330 518T302 809T291 1093Q291 1144 291 1153T296 1164Q298 1165 366 1165Q409 1165 411 1164Q415 1163 416 1157T417 1119Q417 529 517 109T833 -617Q843 -631 843 -635Z"></path><path stroke-width="0" id="E276-MJSZ4-239C" d="M413 -9Q412 -9 407 -9T388 -10T354 -10Q300 -10 297 -9Q294 -8 293 -5Q291 5 291 127V300Q291 602 292 605L296 609Q298 610 366 610Q382 610 392 610T407 610T412 609Q416 609 416 592T417 473V127Q417 -9 413 -9Z"></path><path stroke-width="0" id="E276-MJSZ4-239E" d="M31 1143Q31 1154 49 1154H59Q72 1154 75 1152T89 1136Q190 1013 269 873T401 585T489 294T544 -8T572 -299T583 -583Q583 -634 583 -643T577 -654Q575 -655 508 -655Q465 -655 463 -654Q459 -653 458 -647T457 -609Q457 -58 371 340T100 1037Q87 1059 61 1098T31 1143Z"></path><path stroke-width="0" id="E276-MJSZ4-23A0" d="M56 -644H50Q31 -644 31 -635Q31 -632 37 -622Q69 -579 100 -527Q286 -228 371 170T457 1119Q457 1161 462 1164Q464 1165 520 1165Q575 1165 577 1164Q582 1162 582 1153T583 1093Q581 910 570 752T527 400T442 40T300 -303T89 -626Q78 -640 75 -642T61 -644H56Z"></path><path stroke-width="0" id="E276-MJSZ4-239F" d="M579 -9Q578 -9 573 -9T554 -10T520 -10Q466 -10 463 -9Q460 -8 459 -5Q457 5 457 127V300Q457 602 458 605L462 609Q464 610 532 610Q548 610 558 610T573 610T578 609Q582 609 582 592T583 473V127Q583 -9 579 -9Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><g transform="translate(41043,0)"><g id="mjx-eqn-5"><use xlink:href="#E276-MJMAIN-28"></use><use xlink:href="#E276-MJMAIN-35" x="389" y="0"></use><use xlink:href="#E276-MJMAIN-29" x="889" y="0"></use></g></g><g transform="translate(16539,0)"><g transform="translate(-19,0)"><g transform="translate(0,2563)"><use xlink:href="#E276-MJSZ4-239B" x="0" y="-1155"></use><g transform="translate(0,-2847.787745031177) scale(1,1.783258598411766)"><use xlink:href="#E276-MJSZ4-239C"></use></g><use xlink:href="#E276-MJSZ4-239D" x="0" y="-3983"></use></g><g transform="translate(875,0)"><use xlink:href="#E276-MJMAIN-30" x="0" y="0"></use><use xlink:href="#E276-MJMAIN-2C" x="500" y="0"></use><use xlink:href="#E276-MJMAIN-22EF" x="944" y="0"></use><use xlink:href="#E276-MJMAIN-2C" x="2283" y="0"></use><use xlink:href="#E276-MJMAIN-30" x="2727" y="0"></use><use xlink:href="#E276-MJMAIN-2C" x="3227" y="0"></use><use xlink:href="#E276-MJMAIN-31" x="3672" y="0"></use><use xlink:href="#E276-MJMAIN-2C" x="4172" y="0"></use><use xlink:href="#E276-MJMAIN-30" x="4617" y="0"></use><use xlink:href="#E276-MJMAIN-2C" x="5117" y="0"></use><use xlink:href="#E276-MJMAIN-22EF" x="5562" y="0"></use><use xlink:href="#E276-MJMAIN-2C" x="6900" y="0"></use><use xlink:href="#E276-MJMAIN-30" x="7345" y="0"></use><g transform="translate(3262,-951)"><use transform="scale(0.849)" xlink:href="#E276-MJMAIN-2191" x="528" y="0"></use><g transform="translate(0,-741)"><use transform="scale(1.035)" xlink:href="#E276-MJMATHI-73" x="0" y="0"></use><use transform="scale(1.035)" xlink:href="#E276-MJMAIN-2C" x="469" y="0"></use><use transform="scale(1.035)" xlink:href="#E276-MJMATHI-61" x="747" y="0"></use></g></g></g><g transform="translate(8720,2563)"><use xlink:href="#E276-MJSZ4-239E" x="0" y="-1154"></use><g transform="translate(0,-2847.7700316656374) scale(1,1.7848689043698978)"><use xlink:href="#E276-MJSZ4-239F"></use></g><use xlink:href="#E276-MJSZ4-23A0" x="0" y="-3983"></use></g></g></g></g></svg></span></div><script type="math/tex; mode=display" id="MathJax-Element-249">\left(\underset{\underset{\huge{s,a}}{\large\uparrow}}{0, \cdots, 0, 1, 0, \cdots, 0}\right)</script></div></div><p><span>即在某个的状态动作对处为 1 ，其他都为 0 。这样所有向量的线性组合就是整个动作价值函数，线性组合系数的值就是动作价值函数的值。</span></p><p><span>在使用线性近似的情况下，还可以使用线性最小二乘来进行策略评估。线性最小二乘是一种批处理（batch）方法，它每次针对多个经验样本，试图找到在整个样本集上最优的估计。将线性最小二乘用于回合更新，可以得到</span><strong><span>线性最小二乘回合更新</span></strong><span>（Linear Least Square Monte Carlo, Linear LSMC），它试图最小化目标：</span></p><div contenteditable="false" spellcheck="false" class="mathjax-block md-end-block md-math-block md-rawblock" id="mathjax-n34" cid="n34" mdtype="math_block"><div class="md-rawblock-container md-math-container" tabindex="-1"><div class="MathJax_SVG_Display"><span class="MathJax_SVG" id="MathJax-Element-250-Frame" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="98.296ex" height="5.218ex" viewBox="0 -1372.6 42321.7 2246.8" role="img" focusable="false" style="vertical-align: -1.878ex; margin-bottom: -0.152ex; max-width: 100%;"><defs><path stroke-width="0" id="E277-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E277-MJMAIN-36" d="M42 313Q42 476 123 571T303 666Q372 666 402 630T432 550Q432 525 418 510T379 495Q356 495 341 509T326 548Q326 592 373 601Q351 623 311 626Q240 626 194 566Q147 500 147 364L148 360Q153 366 156 373Q197 433 263 433H267Q313 433 348 414Q372 400 396 374T435 317Q456 268 456 210V192Q456 169 451 149Q440 90 387 34T253 -22Q225 -22 199 -14T143 16T92 75T56 172T42 313ZM257 397Q227 397 205 380T171 335T154 278T148 216Q148 133 160 97T198 39Q222 21 251 21Q302 21 329 59Q342 77 347 104T352 209Q352 289 347 316T329 361Q302 397 257 397Z"></path><path stroke-width="0" id="E277-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E277-MJMATHI-4C" d="M228 637Q194 637 192 641Q191 643 191 649Q191 673 202 682Q204 683 217 683Q271 680 344 680Q485 680 506 683H518Q524 677 524 674T522 656Q517 641 513 637H475Q406 636 394 628Q387 624 380 600T313 336Q297 271 279 198T252 88L243 52Q243 48 252 48T311 46H328Q360 46 379 47T428 54T478 72T522 106T564 161Q580 191 594 228T611 270Q616 273 628 273H641Q647 264 647 262T627 203T583 83T557 9Q555 4 553 3T537 0T494 -1Q483 -1 418 -1T294 0H116Q32 0 32 10Q32 17 34 24Q39 43 44 45Q48 46 59 46H65Q92 46 125 49Q139 52 144 61Q147 65 216 339T285 628Q285 635 228 637Z"></path><path stroke-width="0" id="E277-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E277-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E277-MJSZ2-2211" d="M60 948Q63 950 665 950H1267L1325 815Q1384 677 1388 669H1348L1341 683Q1320 724 1285 761Q1235 809 1174 838T1033 881T882 898T699 902H574H543H251L259 891Q722 258 724 252Q725 250 724 246Q721 243 460 -56L196 -356Q196 -357 407 -357Q459 -357 548 -357T676 -358Q812 -358 896 -353T1063 -332T1204 -283T1307 -196Q1328 -170 1348 -124H1388Q1388 -125 1381 -145T1356 -210T1325 -294L1267 -449L666 -450Q64 -450 61 -448Q55 -446 55 -439Q55 -437 57 -433L590 177Q590 178 557 222T452 366T322 544L56 909L55 924Q55 945 60 948Z"></path><path stroke-width="0" id="E277-MJMATHI-74" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path><path stroke-width="0" id="E277-MJMAIN-5B" d="M118 -250V750H255V710H158V-210H255V-250H118Z"></path><path stroke-width="0" id="E277-MJMATHI-47" d="M50 252Q50 367 117 473T286 641T490 704Q580 704 633 653Q642 643 648 636T656 626L657 623Q660 623 684 649Q691 655 699 663T715 679T725 690L740 705H746Q760 705 760 698Q760 694 728 561Q692 422 692 421Q690 416 687 415T669 413H653Q647 419 647 422Q647 423 648 429T650 449T651 481Q651 552 619 605T510 659Q492 659 471 656T418 643T357 615T294 567T236 496T189 394T158 260Q156 242 156 221Q156 173 170 136T206 79T256 45T308 28T353 24Q407 24 452 47T514 106Q517 114 529 161T541 214Q541 222 528 224T468 227H431Q425 233 425 235T427 254Q431 267 437 273H454Q494 271 594 271Q634 271 659 271T695 272T707 272Q721 272 721 263Q721 261 719 249Q714 230 709 228Q706 227 694 227Q674 227 653 224Q646 221 643 215T629 164Q620 131 614 108Q589 6 586 3Q584 1 581 1Q571 1 553 21T530 52Q530 53 528 52T522 47Q448 -22 322 -22Q201 -22 126 55T50 252Z"></path><path stroke-width="0" id="E277-MJMAIN-2212" d="M84 237T84 250T98 270H679Q694 262 694 250T679 230H98Q84 237 84 250Z"></path><path stroke-width="0" id="E277-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E277-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E277-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E277-MJMATHI-41" d="M208 74Q208 50 254 46Q272 46 272 35Q272 34 270 22Q267 8 264 4T251 0Q249 0 239 0T205 1T141 2Q70 2 50 0H42Q35 7 35 11Q37 38 48 46H62Q132 49 164 96Q170 102 345 401T523 704Q530 716 547 716H555H572Q578 707 578 706L606 383Q634 60 636 57Q641 46 701 46Q726 46 726 36Q726 34 723 22Q720 7 718 4T704 0Q701 0 690 0T651 1T578 2Q484 2 455 0H443Q437 6 437 9T439 27Q443 40 445 43L449 46H469Q523 49 533 63L521 213H283L249 155Q208 86 208 74ZM516 260Q516 271 504 416T490 562L463 519Q447 492 400 412L310 260L413 259Q516 259 516 260Z"></path><path stroke-width="0" id="E277-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E277-MJMAIN-5D" d="M22 710V750H159V-250H22V-210H119V710H22Z"></path><path stroke-width="0" id="E277-MJMAIN-32" d="M109 429Q82 429 66 447T50 491Q50 562 103 614T235 666Q326 666 387 610T449 465Q449 422 429 383T381 315T301 241Q265 210 201 149L142 93L218 92Q375 92 385 97Q392 99 409 186V189H449V186Q448 183 436 95T421 3V0H50V19V31Q50 38 56 46T86 81Q115 113 136 137Q145 147 170 174T204 211T233 244T261 278T284 308T305 340T320 369T333 401T340 431T343 464Q343 527 309 573T212 619Q179 619 154 602T119 569T109 550Q109 549 114 549Q132 549 151 535T170 489Q170 464 154 447T109 429Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><g transform="translate(41043,0)"><g id="mjx-eqn-6" transform="translate(0,358)"><use xlink:href="#E277-MJMAIN-28"></use><use xlink:href="#E277-MJMAIN-36" x="389" y="0"></use><use xlink:href="#E277-MJMAIN-29" x="889" y="0"></use></g></g><g transform="translate(14600,0)"><g transform="translate(-19,0)"><g transform="translate(0,358)"><use xlink:href="#E277-MJMATHI-4C" x="0" y="0"></use><use xlink:href="#E277-MJMAIN-28" x="681" y="0"></use><use xlink:href="#E277-MJMAINB-77" x="1070" y="0"></use><use xlink:href="#E277-MJMAIN-29" x="1901" y="0"></use><use xlink:href="#E277-MJMAIN-3D" x="2567" y="0"></use><g transform="translate(3623,0)"><use xlink:href="#E277-MJSZ2-2211" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E277-MJMATHI-74" x="840" y="-1499"></use></g><use xlink:href="#E277-MJMAIN-5B" x="5067" y="0"></use><g transform="translate(5345,0)"><use xlink:href="#E277-MJMATHI-47" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E277-MJMATHI-74" x="1111" y="-213"></use></g><use xlink:href="#E277-MJMAIN-2212" x="6709" y="0"></use><use xlink:href="#E277-MJMATHI-71" x="7709" y="0"></use><use xlink:href="#E277-MJMAIN-28" x="8169" y="0"></use><g transform="translate(8558,0)"><use xlink:href="#E277-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E277-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E277-MJMAIN-2C" x="9526" y="0"></use><g transform="translate(9971,0)"><use xlink:href="#E277-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E277-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E277-MJMAIN-3B" x="11076" y="0"></use><use xlink:href="#E277-MJMAINB-77" x="11521" y="0"></use><use xlink:href="#E277-MJMAIN-29" x="12352" y="0"></use><g transform="translate(12741,0)"><use xlink:href="#E277-MJMAIN-5D" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E277-MJMAIN-32" x="393" y="583"></use></g></g></g></g></g></svg></span></div><script type="math/tex; mode=display" id="MathJax-Element-250">L(\bold w) = \sum_t [G_t - q(S_t, A_t; \bold w)]^2</script></div></div><p><span>在线性近似的情况下，其梯度为：</span></p><div contenteditable="false" spellcheck="false" class="mathjax-block md-end-block md-math-block md-rawblock" id="mathjax-n36" cid="n36" mdtype="math_block"><div class="md-rawblock-container md-math-container" tabindex="-1"><div class="MathJax_SVG_Display"><span class="MathJax_SVG" id="MathJax-Element-251-Frame" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="98.296ex" height="16.987ex" viewBox="0 -3906.1 42321.7 7313.7" role="img" focusable="false" style="vertical-align: -7.915ex; max-width: 100%;"><defs><path stroke-width="0" id="E278-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E278-MJMAIN-37" d="M55 458Q56 460 72 567L88 674Q88 676 108 676H128V672Q128 662 143 655T195 646T364 644H485V605L417 512Q408 500 387 472T360 435T339 403T319 367T305 330T292 284T284 230T278 162T275 80Q275 66 275 52T274 28V19Q270 2 255 -10T221 -22Q210 -22 200 -19T179 0T168 40Q168 198 265 368Q285 400 349 489L395 552H302Q128 552 119 546Q113 543 108 522T98 479L95 458V455H55V458Z"></path><path stroke-width="0" id="E278-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E278-MJSZ2-2211" d="M60 948Q63 950 665 950H1267L1325 815Q1384 677 1388 669H1348L1341 683Q1320 724 1285 761Q1235 809 1174 838T1033 881T882 898T699 902H574H543H251L259 891Q722 258 724 252Q725 250 724 246Q721 243 460 -56L196 -356Q196 -357 407 -357Q459 -357 548 -357T676 -358Q812 -358 896 -353T1063 -332T1204 -283T1307 -196Q1328 -170 1348 -124H1388Q1388 -125 1381 -145T1356 -210T1325 -294L1267 -449L666 -450Q64 -450 61 -448Q55 -446 55 -439Q55 -437 57 -433L590 177Q590 178 557 222T452 366T322 544L56 909L55 924Q55 945 60 948Z"></path><path stroke-width="0" id="E278-MJMATHI-74" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path><path stroke-width="0" id="E278-MJMAIN-5B" d="M118 -250V750H255V710H158V-210H255V-250H118Z"></path><path stroke-width="0" id="E278-MJMATHI-47" d="M50 252Q50 367 117 473T286 641T490 704Q580 704 633 653Q642 643 648 636T656 626L657 623Q660 623 684 649Q691 655 699 663T715 679T725 690L740 705H746Q760 705 760 698Q760 694 728 561Q692 422 692 421Q690 416 687 415T669 413H653Q647 419 647 422Q647 423 648 429T650 449T651 481Q651 552 619 605T510 659Q492 659 471 656T418 643T357 615T294 567T236 496T189 394T158 260Q156 242 156 221Q156 173 170 136T206 79T256 45T308 28T353 24Q407 24 452 47T514 106Q517 114 529 161T541 214Q541 222 528 224T468 227H431Q425 233 425 235T427 254Q431 267 437 273H454Q494 271 594 271Q634 271 659 271T695 272T707 272Q721 272 721 263Q721 261 719 249Q714 230 709 228Q706 227 694 227Q674 227 653 224Q646 221 643 215T629 164Q620 131 614 108Q589 6 586 3Q584 1 581 1Q571 1 553 21T530 52Q530 53 528 52T522 47Q448 -22 322 -22Q201 -22 126 55T50 252Z"></path><path stroke-width="0" id="E278-MJMAIN-2212" d="M84 237T84 250T98 270H679Q694 262 694 250T679 230H98Q84 237 84 250Z"></path><path stroke-width="0" id="E278-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E278-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E278-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E278-MJMATHI-41" d="M208 74Q208 50 254 46Q272 46 272 35Q272 34 270 22Q267 8 264 4T251 0Q249 0 239 0T205 1T141 2Q70 2 50 0H42Q35 7 35 11Q37 38 48 46H62Q132 49 164 96Q170 102 345 401T523 704Q530 716 547 716H555H572Q578 707 578 706L606 383Q634 60 636 57Q641 46 701 46Q726 46 726 36Q726 34 723 22Q720 7 718 4T704 0Q701 0 690 0T651 1T578 2Q484 2 455 0H443Q437 6 437 9T439 27Q443 40 445 43L449 46H469Q523 49 533 63L521 213H283L249 155Q208 86 208 74ZM516 260Q516 271 504 416T490 562L463 519Q447 492 400 412L310 260L413 259Q516 259 516 260Z"></path><path stroke-width="0" id="E278-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E278-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E278-MJMAIN-5D" d="M22 710V750H159V-250H22V-210H119V710H22Z"></path><path stroke-width="0" id="E278-MJMAIN-2207" d="M46 676Q46 679 51 683H781Q786 679 786 676Q786 674 617 326T444 -26Q439 -33 416 -33T388 -26Q385 -22 216 326T46 676ZM697 596Q697 597 445 597T193 596Q195 591 319 336T445 80L697 596Z"></path><path stroke-width="0" id="E278-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E278-MJMAINB-78" d="M227 0Q212 3 121 3Q40 3 28 0H21V62H117L245 213L109 382H26V444H34Q49 441 143 441Q247 441 265 444H274V382H246L281 339Q315 297 316 297Q320 297 354 341L389 382H352V444H360Q375 441 466 441Q547 441 559 444H566V382H471L355 246L504 63L545 62H586V0H578Q563 3 469 3Q365 3 347 0H338V62H366Q366 63 326 112T285 163L198 63L217 62H235V0H227Z"></path><path stroke-width="0" id="E278-MJMATHI-61" d="M33 157Q33 258 109 349T280 441Q331 441 370 392Q386 422 416 422Q429 422 439 414T449 394Q449 381 412 234T374 68Q374 43 381 35T402 26Q411 27 422 35Q443 55 463 131Q469 151 473 152Q475 153 483 153H487Q506 153 506 144Q506 138 501 117T481 63T449 13Q436 0 417 -8Q409 -10 393 -10Q359 -10 336 5T306 36L300 51Q299 52 296 50Q294 48 292 46Q233 -10 172 -10Q117 -10 75 30T33 157ZM351 328Q351 334 346 350T323 385T277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q217 26 254 59T298 110Q300 114 325 217T351 328Z"></path><path stroke-width="0" id="E278-MJMATHI-54" d="M40 437Q21 437 21 445Q21 450 37 501T71 602L88 651Q93 669 101 677H569H659Q691 677 697 676T704 667Q704 661 687 553T668 444Q668 437 649 437Q640 437 637 437T631 442L629 445Q629 451 635 490T641 551Q641 586 628 604T573 629Q568 630 515 631Q469 631 457 630T439 622Q438 621 368 343T298 60Q298 48 386 46Q418 46 427 45T436 36Q436 31 433 22Q429 4 424 1L422 0Q419 0 415 0Q410 0 363 1T228 2Q99 2 64 0H49Q43 6 43 9T45 27Q49 40 55 46H83H94Q174 46 189 55Q190 56 191 56Q196 59 201 76T241 233Q258 301 269 344Q339 619 339 625Q339 630 310 630H279Q212 630 191 624Q146 614 121 583T67 467Q60 445 57 441T43 437H40Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><g transform="translate(41043,0)"><g id="mjx-eqn-7" transform="translate(0,2906)"><use xlink:href="#E278-MJMAIN-28"></use><use xlink:href="#E278-MJMAIN-37" x="389" y="0"></use><use xlink:href="#E278-MJMAIN-29" x="889" y="0"></use></g></g><g transform="translate(11058,0)"><g transform="translate(-19,0)"><g transform="translate(0,2906)"><use xlink:href="#E278-MJSZ2-2211" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E278-MJMATHI-74" x="840" y="-1499"></use><use xlink:href="#E278-MJMAIN-5B" x="1444" y="0"></use><g transform="translate(1722,0)"><use xlink:href="#E278-MJMATHI-47" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E278-MJMATHI-74" x="1111" y="-213"></use></g><use xlink:href="#E278-MJMAIN-2212" x="3085" y="0"></use><use xlink:href="#E278-MJMATHI-71" x="4085" y="0"></use><use xlink:href="#E278-MJMAIN-28" x="4545" y="0"></use><g transform="translate(4934,0)"><use xlink:href="#E278-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E278-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E278-MJMAIN-2C" x="5902" y="0"></use><g transform="translate(6347,0)"><use xlink:href="#E278-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E278-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E278-MJMAIN-3B" x="7452" y="0"></use><use xlink:href="#E278-MJMAINB-77" x="7897" y="0"></use><use xlink:href="#E278-MJMAIN-29" x="8728" y="0"></use><use xlink:href="#E278-MJMAIN-5D" x="9117" y="0"></use><use xlink:href="#E278-MJMAIN-2207" x="9395" y="0"></use><use xlink:href="#E278-MJMATHI-71" x="10228" y="0"></use><use xlink:href="#E278-MJMAIN-28" x="10688" y="0"></use><g transform="translate(11077,0)"><use xlink:href="#E278-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E278-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E278-MJMAIN-2C" x="12045" y="0"></use><g transform="translate(12490,0)"><use xlink:href="#E278-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E278-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E278-MJMAIN-3B" x="13595" y="0"></use><use xlink:href="#E278-MJMAINB-77" x="14040" y="0"></use><use xlink:href="#E278-MJMAIN-29" x="14871" y="0"></use><g transform="translate(0,-2548)"><use xlink:href="#E278-MJMAIN-3D" x="0" y="0"></use><g transform="translate(1055,0)"><use xlink:href="#E278-MJSZ2-2211" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E278-MJMATHI-74" x="840" y="-1499"></use></g><use xlink:href="#E278-MJMAIN-5B" x="2499" y="0"></use><g transform="translate(2777,0)"><use xlink:href="#E278-MJMATHI-47" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E278-MJMATHI-74" x="1111" y="-213"></use></g><use xlink:href="#E278-MJMAIN-2212" x="4141" y="0"></use><use xlink:href="#E278-MJMAIN-28" x="5141" y="0"></use><use xlink:href="#E278-MJMAINB-78" x="5530" y="0"></use><use xlink:href="#E278-MJMAIN-28" x="6137" y="0"></use><g transform="translate(6526,0)"><use xlink:href="#E278-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E278-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E278-MJMAIN-2C" x="7494" y="0"></use><g transform="translate(7939,0)"><use xlink:href="#E278-MJMATHI-61" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E278-MJMATHI-54" x="748" y="-213"></use></g><use xlink:href="#E278-MJMAIN-29" x="9066" y="0"></use><g transform="translate(9455,0)"><use xlink:href="#E278-MJMAIN-29" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E278-MJMATHI-54" x="550" y="583"></use></g><use xlink:href="#E278-MJMAINB-77" x="10442" y="0"></use><use xlink:href="#E278-MJMAIN-5D" x="11273" y="0"></use><use xlink:href="#E278-MJMAINB-78" x="11551" y="0"></use><use xlink:href="#E278-MJMAIN-28" x="12158" y="0"></use><g transform="translate(12547,0)"><use xlink:href="#E278-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E278-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E278-MJMAIN-2C" x="13515" y="0"></use><g transform="translate(13959,0)"><use xlink:href="#E278-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E278-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E278-MJMAIN-29" x="15065" y="0"></use></g><g transform="translate(0,-5096)"><use xlink:href="#E278-MJMAIN-3D" x="0" y="0"></use><g transform="translate(1055,0)"><use xlink:href="#E278-MJSZ2-2211" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E278-MJMATHI-74" x="840" y="-1499"></use></g><g transform="translate(2666,0)"><use xlink:href="#E278-MJMATHI-47" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E278-MJMATHI-74" x="1111" y="-213"></use></g><use xlink:href="#E278-MJMAINB-78" x="3807" y="0"></use><use xlink:href="#E278-MJMAIN-28" x="4414" y="0"></use><g transform="translate(4803,0)"><use xlink:href="#E278-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E278-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E278-MJMAIN-2C" x="5771" y="0"></use><g transform="translate(6216,0)"><use xlink:href="#E278-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E278-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E278-MJMAIN-29" x="7321" y="0"></use><use xlink:href="#E278-MJMAIN-2212" x="7933" y="0"></use><g transform="translate(8933,0)"><use xlink:href="#E278-MJSZ2-2211" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E278-MJMATHI-74" x="840" y="-1499"></use></g><use xlink:href="#E278-MJMAINB-78" x="10544" y="0"></use><use xlink:href="#E278-MJMAIN-28" x="11151" y="0"></use><g transform="translate(11540,0)"><use xlink:href="#E278-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E278-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E278-MJMAIN-2C" x="12508" y="0"></use><g transform="translate(12952,0)"><use xlink:href="#E278-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E278-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E278-MJMAIN-29" x="14058" y="0"></use><use xlink:href="#E278-MJMAIN-28" x="14447" y="0"></use><use xlink:href="#E278-MJMAINB-78" x="14836" y="0"></use><use xlink:href="#E278-MJMAIN-28" x="15443" y="0"></use><g transform="translate(15832,0)"><use xlink:href="#E278-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E278-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E278-MJMAIN-2C" x="16800" y="0"></use><g transform="translate(17245,0)"><use xlink:href="#E278-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E278-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E278-MJMAIN-29" x="18350" y="0"></use><g transform="translate(18739,0)"><use xlink:href="#E278-MJMAIN-29" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E278-MJMATHI-54" x="550" y="583"></use></g><use xlink:href="#E278-MJMAINB-77" x="19726" y="0"></use></g></g></g></g></g></svg></span></div><script type="math/tex; mode=display" id="MathJax-Element-251">\sum_t [G_t - q(S_t,A_t;\bold w)]\nabla q(S_t,A_t;\bold w) \\
 = \sum_t [G_t - (\bold x(S_t,a_T))^T\bold w]\bold x(S_t,A_t) \\
 = \sum_t G_t\bold x(S_t,A_t) - \sum_t \bold x(S_t,A_t)(\bold x(S_t,A_t))^T\bold w</script></div></div><p><span>将待求权重 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="7.314ex" height="1.746ex" viewBox="0 -500.4 3149.2 751.6" role="img" focusable="false" style="vertical-align: -0.583ex;"><defs><path stroke-width="0" id="E312-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E312-MJMATHI-4C" d="M228 637Q194 637 192 641Q191 643 191 649Q191 673 202 682Q204 683 217 683Q271 680 344 680Q485 680 506 683H518Q524 677 524 674T522 656Q517 641 513 637H475Q406 636 394 628Q387 624 380 600T313 336Q297 271 279 198T252 88L243 52Q243 48 252 48T311 46H328Q360 46 379 47T428 54T478 72T522 106T564 161Q580 191 594 228T611 270Q616 273 628 273H641Q647 264 647 262T627 203T583 83T557 9Q555 4 553 3T537 0T494 -1Q483 -1 418 -1T294 0H116Q32 0 32 10Q32 17 34 24Q39 43 44 45Q48 46 59 46H65Q92 46 125 49Q139 52 144 61Q147 65 216 339T285 628Q285 635 228 637Z"></path><path stroke-width="0" id="E312-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E312-MJMATHI-4D" d="M289 629Q289 635 232 637Q208 637 201 638T194 648Q194 649 196 659Q197 662 198 666T199 671T201 676T203 679T207 681T212 683T220 683T232 684Q238 684 262 684T307 683Q386 683 398 683T414 678Q415 674 451 396L487 117L510 154Q534 190 574 254T662 394Q837 673 839 675Q840 676 842 678T846 681L852 683H948Q965 683 988 683T1017 684Q1051 684 1051 673Q1051 668 1048 656T1045 643Q1041 637 1008 637Q968 636 957 634T939 623Q936 618 867 340T797 59Q797 55 798 54T805 50T822 48T855 46H886Q892 37 892 35Q892 19 885 5Q880 0 869 0Q864 0 828 1T736 2Q675 2 644 2T609 1Q592 1 592 11Q592 13 594 25Q598 41 602 43T625 46Q652 46 685 49Q699 52 704 61Q706 65 742 207T813 490T848 631L654 322Q458 10 453 5Q451 4 449 3Q444 0 433 0Q418 0 415 7Q413 11 374 317L335 624L267 354Q200 88 200 79Q206 46 272 46H282Q288 41 289 37T286 19Q282 3 278 1Q274 0 267 0Q265 0 255 0T221 1T157 2Q127 2 95 1T58 0Q43 0 39 2T35 11Q35 13 38 25T43 40Q45 46 65 46Q135 46 154 86Q158 92 223 354T289 629Z"></path><path stroke-width="0" id="E312-MJMATHI-43" d="M50 252Q50 367 117 473T286 641T490 704Q580 704 633 653Q642 643 648 636T656 626L657 623Q660 623 684 649Q691 655 699 663T715 679T725 690L740 705H746Q760 705 760 698Q760 694 728 561Q692 422 692 421Q690 416 687 415T669 413H653Q647 419 647 422Q647 423 648 429T650 449T651 481Q651 552 619 605T510 659Q484 659 454 652T382 628T299 572T226 479Q194 422 175 346T156 222Q156 108 232 58Q280 24 350 24Q441 24 512 92T606 240Q610 253 612 255T628 257Q648 257 648 248Q648 243 647 239Q618 132 523 55T319 -22Q206 -22 128 53T50 252Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E312-MJMAINB-77" x="0" y="0"></use><g transform="translate(831,-155)"><use transform="scale(0.707)" xlink:href="#E312-MJMATHI-4C" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E312-MJMATHI-53" x="681" y="0"></use><use transform="scale(0.707)" xlink:href="#E312-MJMATHI-4D" x="1326" y="0"></use><use transform="scale(0.707)" xlink:href="#E312-MJMATHI-43" x="2377" y="0"></use></g></g></svg></span><script type="math/tex">\bold w_{LSMC}</script><span> 代入上式并令其等于零，则有：</span></p><div contenteditable="false" spellcheck="false" class="mathjax-block md-end-block md-math-block md-rawblock" id="mathjax-n38" cid="n38" mdtype="math_block"><div class="md-rawblock-container md-math-container" tabindex="-1"><div class="MathJax_SVG_Display"><span class="MathJax_SVG" id="MathJax-Element-252-Frame" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="98.296ex" height="5.218ex" viewBox="0 -1372.6 42321.7 2246.8" role="img" focusable="false" style="vertical-align: -1.878ex; margin-bottom: -0.152ex; max-width: 100%;"><defs><path stroke-width="0" id="E279-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E279-MJMAIN-38" d="M70 417T70 494T124 618T248 666Q319 666 374 624T429 515Q429 485 418 459T392 417T361 389T335 371T324 363L338 354Q352 344 366 334T382 323Q457 264 457 174Q457 95 399 37T249 -22Q159 -22 101 29T43 155Q43 263 172 335L154 348Q133 361 127 368Q70 417 70 494ZM286 386L292 390Q298 394 301 396T311 403T323 413T334 425T345 438T355 454T364 471T369 491T371 513Q371 556 342 586T275 624Q268 625 242 625Q201 625 165 599T128 534Q128 511 141 492T167 463T217 431Q224 426 228 424L286 386ZM250 21Q308 21 350 55T392 137Q392 154 387 169T375 194T353 216T330 234T301 253T274 270Q260 279 244 289T218 306L210 311Q204 311 181 294T133 239T107 157Q107 98 150 60T250 21Z"></path><path stroke-width="0" id="E279-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E279-MJSZ2-2211" d="M60 948Q63 950 665 950H1267L1325 815Q1384 677 1388 669H1348L1341 683Q1320 724 1285 761Q1235 809 1174 838T1033 881T882 898T699 902H574H543H251L259 891Q722 258 724 252Q725 250 724 246Q721 243 460 -56L196 -356Q196 -357 407 -357Q459 -357 548 -357T676 -358Q812 -358 896 -353T1063 -332T1204 -283T1307 -196Q1328 -170 1348 -124H1388Q1388 -125 1381 -145T1356 -210T1325 -294L1267 -449L666 -450Q64 -450 61 -448Q55 -446 55 -439Q55 -437 57 -433L590 177Q590 178 557 222T452 366T322 544L56 909L55 924Q55 945 60 948Z"></path><path stroke-width="0" id="E279-MJMATHI-74" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path><path stroke-width="0" id="E279-MJMATHI-47" d="M50 252Q50 367 117 473T286 641T490 704Q580 704 633 653Q642 643 648 636T656 626L657 623Q660 623 684 649Q691 655 699 663T715 679T725 690L740 705H746Q760 705 760 698Q760 694 728 561Q692 422 692 421Q690 416 687 415T669 413H653Q647 419 647 422Q647 423 648 429T650 449T651 481Q651 552 619 605T510 659Q492 659 471 656T418 643T357 615T294 567T236 496T189 394T158 260Q156 242 156 221Q156 173 170 136T206 79T256 45T308 28T353 24Q407 24 452 47T514 106Q517 114 529 161T541 214Q541 222 528 224T468 227H431Q425 233 425 235T427 254Q431 267 437 273H454Q494 271 594 271Q634 271 659 271T695 272T707 272Q721 272 721 263Q721 261 719 249Q714 230 709 228Q706 227 694 227Q674 227 653 224Q646 221 643 215T629 164Q620 131 614 108Q589 6 586 3Q584 1 581 1Q571 1 553 21T530 52Q530 53 528 52T522 47Q448 -22 322 -22Q201 -22 126 55T50 252Z"></path><path stroke-width="0" id="E279-MJMAINB-78" d="M227 0Q212 3 121 3Q40 3 28 0H21V62H117L245 213L109 382H26V444H34Q49 441 143 441Q247 441 265 444H274V382H246L281 339Q315 297 316 297Q320 297 354 341L389 382H352V444H360Q375 441 466 441Q547 441 559 444H566V382H471L355 246L504 63L545 62H586V0H578Q563 3 469 3Q365 3 347 0H338V62H366Q366 63 326 112T285 163L198 63L217 62H235V0H227Z"></path><path stroke-width="0" id="E279-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E279-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E279-MJMATHI-41" d="M208 74Q208 50 254 46Q272 46 272 35Q272 34 270 22Q267 8 264 4T251 0Q249 0 239 0T205 1T141 2Q70 2 50 0H42Q35 7 35 11Q37 38 48 46H62Q132 49 164 96Q170 102 345 401T523 704Q530 716 547 716H555H572Q578 707 578 706L606 383Q634 60 636 57Q641 46 701 46Q726 46 726 36Q726 34 723 22Q720 7 718 4T704 0Q701 0 690 0T651 1T578 2Q484 2 455 0H443Q437 6 437 9T439 27Q443 40 445 43L449 46H469Q523 49 533 63L521 213H283L249 155Q208 86 208 74ZM516 260Q516 271 504 416T490 562L463 519Q447 492 400 412L310 260L413 259Q516 259 516 260Z"></path><path stroke-width="0" id="E279-MJMAIN-2212" d="M84 237T84 250T98 270H679Q694 262 694 250T679 230H98Q84 237 84 250Z"></path><path stroke-width="0" id="E279-MJMATHI-54" d="M40 437Q21 437 21 445Q21 450 37 501T71 602L88 651Q93 669 101 677H569H659Q691 677 697 676T704 667Q704 661 687 553T668 444Q668 437 649 437Q640 437 637 437T631 442L629 445Q629 451 635 490T641 551Q641 586 628 604T573 629Q568 630 515 631Q469 631 457 630T439 622Q438 621 368 343T298 60Q298 48 386 46Q418 46 427 45T436 36Q436 31 433 22Q429 4 424 1L422 0Q419 0 415 0Q410 0 363 1T228 2Q99 2 64 0H49Q43 6 43 9T45 27Q49 40 55 46H83H94Q174 46 189 55Q190 56 191 56Q196 59 201 76T241 233Q258 301 269 344Q339 619 339 625Q339 630 310 630H279Q212 630 191 624Q146 614 121 583T67 467Q60 445 57 441T43 437H40Z"></path><path stroke-width="0" id="E279-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E279-MJMATHI-4C" d="M228 637Q194 637 192 641Q191 643 191 649Q191 673 202 682Q204 683 217 683Q271 680 344 680Q485 680 506 683H518Q524 677 524 674T522 656Q517 641 513 637H475Q406 636 394 628Q387 624 380 600T313 336Q297 271 279 198T252 88L243 52Q243 48 252 48T311 46H328Q360 46 379 47T428 54T478 72T522 106T564 161Q580 191 594 228T611 270Q616 273 628 273H641Q647 264 647 262T627 203T583 83T557 9Q555 4 553 3T537 0T494 -1Q483 -1 418 -1T294 0H116Q32 0 32 10Q32 17 34 24Q39 43 44 45Q48 46 59 46H65Q92 46 125 49Q139 52 144 61Q147 65 216 339T285 628Q285 635 228 637Z"></path><path stroke-width="0" id="E279-MJMATHI-4D" d="M289 629Q289 635 232 637Q208 637 201 638T194 648Q194 649 196 659Q197 662 198 666T199 671T201 676T203 679T207 681T212 683T220 683T232 684Q238 684 262 684T307 683Q386 683 398 683T414 678Q415 674 451 396L487 117L510 154Q534 190 574 254T662 394Q837 673 839 675Q840 676 842 678T846 681L852 683H948Q965 683 988 683T1017 684Q1051 684 1051 673Q1051 668 1048 656T1045 643Q1041 637 1008 637Q968 636 957 634T939 623Q936 618 867 340T797 59Q797 55 798 54T805 50T822 48T855 46H886Q892 37 892 35Q892 19 885 5Q880 0 869 0Q864 0 828 1T736 2Q675 2 644 2T609 1Q592 1 592 11Q592 13 594 25Q598 41 602 43T625 46Q652 46 685 49Q699 52 704 61Q706 65 742 207T813 490T848 631L654 322Q458 10 453 5Q451 4 449 3Q444 0 433 0Q418 0 415 7Q413 11 374 317L335 624L267 354Q200 88 200 79Q206 46 272 46H282Q288 41 289 37T286 19Q282 3 278 1Q274 0 267 0Q265 0 255 0T221 1T157 2Q127 2 95 1T58 0Q43 0 39 2T35 11Q35 13 38 25T43 40Q45 46 65 46Q135 46 154 86Q158 92 223 354T289 629Z"></path><path stroke-width="0" id="E279-MJMATHI-43" d="M50 252Q50 367 117 473T286 641T490 704Q580 704 633 653Q642 643 648 636T656 626L657 623Q660 623 684 649Q691 655 699 663T715 679T725 690L740 705H746Q760 705 760 698Q760 694 728 561Q692 422 692 421Q690 416 687 415T669 413H653Q647 419 647 422Q647 423 648 429T650 449T651 481Q651 552 619 605T510 659Q484 659 454 652T382 628T299 572T226 479Q194 422 175 346T156 222Q156 108 232 58Q280 24 350 24Q441 24 512 92T606 240Q610 253 612 255T628 257Q648 257 648 248Q648 243 647 239Q618 132 523 55T319 -22Q206 -22 128 53T50 252Z"></path><path stroke-width="0" id="E279-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E279-MJMAINB-30" d="M266 654H280H282Q500 654 524 418Q529 370 529 320Q529 125 456 52Q397 -10 287 -10Q110 -10 63 154Q45 212 45 316Q45 504 113 585Q140 618 185 636T266 654ZM374 548Q347 604 286 604Q247 604 218 575Q197 552 193 511T188 311Q188 159 196 116Q202 87 225 64T287 41Q339 41 367 87Q379 107 382 152T386 329Q386 518 374 548Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><g transform="translate(41043,0)"><g id="mjx-eqn-8" transform="translate(0,358)"><use xlink:href="#E279-MJMAIN-28"></use><use xlink:href="#E279-MJMAIN-38" x="389" y="0"></use><use xlink:href="#E279-MJMAIN-29" x="889" y="0"></use></g></g><g transform="translate(9472,0)"><g transform="translate(-19,0)"><g transform="translate(0,358)"><use xlink:href="#E279-MJSZ2-2211" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E279-MJMATHI-74" x="840" y="-1499"></use><g transform="translate(1610,0)"><use xlink:href="#E279-MJMATHI-47" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E279-MJMATHI-74" x="1111" y="-213"></use></g><use xlink:href="#E279-MJMAINB-78" x="2751" y="0"></use><use xlink:href="#E279-MJMAIN-28" x="3358" y="0"></use><g transform="translate(3747,0)"><use xlink:href="#E279-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E279-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E279-MJMAIN-2C" x="4716" y="0"></use><g transform="translate(5160,0)"><use xlink:href="#E279-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E279-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E279-MJMAIN-29" x="6266" y="0"></use><use xlink:href="#E279-MJMAIN-2212" x="6877" y="0"></use><g transform="translate(7877,0)"><use xlink:href="#E279-MJSZ2-2211" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E279-MJMATHI-74" x="840" y="-1499"></use></g><use xlink:href="#E279-MJMAINB-78" x="9488" y="0"></use><use xlink:href="#E279-MJMAIN-28" x="10095" y="0"></use><g transform="translate(10484,0)"><use xlink:href="#E279-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E279-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E279-MJMAIN-2C" x="11452" y="0"></use><g transform="translate(11897,0)"><use xlink:href="#E279-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E279-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E279-MJMAIN-29" x="13002" y="0"></use><use xlink:href="#E279-MJMAIN-28" x="13391" y="0"></use><use xlink:href="#E279-MJMAINB-78" x="13780" y="0"></use><use xlink:href="#E279-MJMAIN-28" x="14387" y="0"></use><g transform="translate(14776,0)"><use xlink:href="#E279-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E279-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E279-MJMAIN-2C" x="15744" y="0"></use><g transform="translate(16189,0)"><use xlink:href="#E279-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E279-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E279-MJMAIN-29" x="17294" y="0"></use><g transform="translate(17683,0)"><use xlink:href="#E279-MJMAIN-29" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E279-MJMATHI-54" x="550" y="583"></use></g><g transform="translate(18670,0)"><use xlink:href="#E279-MJMAINB-77" x="0" y="0"></use><g transform="translate(831,-155)"><use transform="scale(0.707)" xlink:href="#E279-MJMATHI-4C" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E279-MJMATHI-53" x="681" y="0"></use><use transform="scale(0.707)" xlink:href="#E279-MJMATHI-4D" x="1326" y="0"></use><use transform="scale(0.707)" xlink:href="#E279-MJMATHI-43" x="2377" y="0"></use></g></g><use xlink:href="#E279-MJMAIN-3D" x="22097" y="0"></use><use xlink:href="#E279-MJMAINB-30" x="23153" y="0"></use></g></g></g></g></svg></span></div><script type="math/tex; mode=display" id="MathJax-Element-252">\sum_t G_t\bold x(S_t,A_t) - \sum_t \bold x(S_t,A_t)(\bold x(S_t,A_t))^T\bold w_{LSMC} = \bold 0</script></div></div><p><span>求解该线性方程组可得：</span></p><div contenteditable="false" spellcheck="false" class="mathjax-block md-end-block md-math-block md-rawblock" id="mathjax-n40" cid="n40" mdtype="math_block"><div class="md-rawblock-container md-math-container" tabindex="-1"><div class="MathJax_SVG_Display"><span class="MathJax_SVG" id="MathJax-Element-253-Frame" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="98.296ex" height="7.726ex" viewBox="0 -1912.5 42321.7 3326.6" role="img" focusable="false" style="vertical-align: -3.284ex; max-width: 100%;"><defs><path stroke-width="0" id="E280-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E280-MJMAIN-39" d="M352 287Q304 211 232 211Q154 211 104 270T44 396Q42 412 42 436V444Q42 537 111 606Q171 666 243 666Q245 666 249 666T257 665H261Q273 665 286 663T323 651T370 619T413 560Q456 472 456 334Q456 194 396 97Q361 41 312 10T208 -22Q147 -22 108 7T68 93T121 149Q143 149 158 135T173 96Q173 78 164 65T148 49T135 44L131 43Q131 41 138 37T164 27T206 22H212Q272 22 313 86Q352 142 352 280V287ZM244 248Q292 248 321 297T351 430Q351 508 343 542Q341 552 337 562T323 588T293 615T246 625Q208 625 181 598Q160 576 154 546T147 441Q147 358 152 329T172 282Q197 248 244 248Z"></path><path stroke-width="0" id="E280-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E280-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E280-MJMATHI-4C" d="M228 637Q194 637 192 641Q191 643 191 649Q191 673 202 682Q204 683 217 683Q271 680 344 680Q485 680 506 683H518Q524 677 524 674T522 656Q517 641 513 637H475Q406 636 394 628Q387 624 380 600T313 336Q297 271 279 198T252 88L243 52Q243 48 252 48T311 46H328Q360 46 379 47T428 54T478 72T522 106T564 161Q580 191 594 228T611 270Q616 273 628 273H641Q647 264 647 262T627 203T583 83T557 9Q555 4 553 3T537 0T494 -1Q483 -1 418 -1T294 0H116Q32 0 32 10Q32 17 34 24Q39 43 44 45Q48 46 59 46H65Q92 46 125 49Q139 52 144 61Q147 65 216 339T285 628Q285 635 228 637Z"></path><path stroke-width="0" id="E280-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E280-MJMATHI-4D" d="M289 629Q289 635 232 637Q208 637 201 638T194 648Q194 649 196 659Q197 662 198 666T199 671T201 676T203 679T207 681T212 683T220 683T232 684Q238 684 262 684T307 683Q386 683 398 683T414 678Q415 674 451 396L487 117L510 154Q534 190 574 254T662 394Q837 673 839 675Q840 676 842 678T846 681L852 683H948Q965 683 988 683T1017 684Q1051 684 1051 673Q1051 668 1048 656T1045 643Q1041 637 1008 637Q968 636 957 634T939 623Q936 618 867 340T797 59Q797 55 798 54T805 50T822 48T855 46H886Q892 37 892 35Q892 19 885 5Q880 0 869 0Q864 0 828 1T736 2Q675 2 644 2T609 1Q592 1 592 11Q592 13 594 25Q598 41 602 43T625 46Q652 46 685 49Q699 52 704 61Q706 65 742 207T813 490T848 631L654 322Q458 10 453 5Q451 4 449 3Q444 0 433 0Q418 0 415 7Q413 11 374 317L335 624L267 354Q200 88 200 79Q206 46 272 46H282Q288 41 289 37T286 19Q282 3 278 1Q274 0 267 0Q265 0 255 0T221 1T157 2Q127 2 95 1T58 0Q43 0 39 2T35 11Q35 13 38 25T43 40Q45 46 65 46Q135 46 154 86Q158 92 223 354T289 629Z"></path><path stroke-width="0" id="E280-MJMATHI-43" d="M50 252Q50 367 117 473T286 641T490 704Q580 704 633 653Q642 643 648 636T656 626L657 623Q660 623 684 649Q691 655 699 663T715 679T725 690L740 705H746Q760 705 760 698Q760 694 728 561Q692 422 692 421Q690 416 687 415T669 413H653Q647 419 647 422Q647 423 648 429T650 449T651 481Q651 552 619 605T510 659Q484 659 454 652T382 628T299 572T226 479Q194 422 175 346T156 222Q156 108 232 58Q280 24 350 24Q441 24 512 92T606 240Q610 253 612 255T628 257Q648 257 648 248Q648 243 647 239Q618 132 523 55T319 -22Q206 -22 128 53T50 252Z"></path><path stroke-width="0" id="E280-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E280-MJSZ2-2211" d="M60 948Q63 950 665 950H1267L1325 815Q1384 677 1388 669H1348L1341 683Q1320 724 1285 761Q1235 809 1174 838T1033 881T882 898T699 902H574H543H251L259 891Q722 258 724 252Q725 250 724 246Q721 243 460 -56L196 -356Q196 -357 407 -357Q459 -357 548 -357T676 -358Q812 -358 896 -353T1063 -332T1204 -283T1307 -196Q1328 -170 1348 -124H1388Q1388 -125 1381 -145T1356 -210T1325 -294L1267 -449L666 -450Q64 -450 61 -448Q55 -446 55 -439Q55 -437 57 -433L590 177Q590 178 557 222T452 366T322 544L56 909L55 924Q55 945 60 948Z"></path><path stroke-width="0" id="E280-MJMATHI-74" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path><path stroke-width="0" id="E280-MJMAINB-78" d="M227 0Q212 3 121 3Q40 3 28 0H21V62H117L245 213L109 382H26V444H34Q49 441 143 441Q247 441 265 444H274V382H246L281 339Q315 297 316 297Q320 297 354 341L389 382H352V444H360Q375 441 466 441Q547 441 559 444H566V382H471L355 246L504 63L545 62H586V0H578Q563 3 469 3Q365 3 347 0H338V62H366Q366 63 326 112T285 163L198 63L217 62H235V0H227Z"></path><path stroke-width="0" id="E280-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E280-MJMATHI-41" d="M208 74Q208 50 254 46Q272 46 272 35Q272 34 270 22Q267 8 264 4T251 0Q249 0 239 0T205 1T141 2Q70 2 50 0H42Q35 7 35 11Q37 38 48 46H62Q132 49 164 96Q170 102 345 401T523 704Q530 716 547 716H555H572Q578 707 578 706L606 383Q634 60 636 57Q641 46 701 46Q726 46 726 36Q726 34 723 22Q720 7 718 4T704 0Q701 0 690 0T651 1T578 2Q484 2 455 0H443Q437 6 437 9T439 27Q443 40 445 43L449 46H469Q523 49 533 63L521 213H283L249 155Q208 86 208 74ZM516 260Q516 271 504 416T490 562L463 519Q447 492 400 412L310 260L413 259Q516 259 516 260Z"></path><path stroke-width="0" id="E280-MJMATHI-78" d="M52 289Q59 331 106 386T222 442Q257 442 286 424T329 379Q371 442 430 442Q467 442 494 420T522 361Q522 332 508 314T481 292T458 288Q439 288 427 299T415 328Q415 374 465 391Q454 404 425 404Q412 404 406 402Q368 386 350 336Q290 115 290 78Q290 50 306 38T341 26Q378 26 414 59T463 140Q466 150 469 151T485 153H489Q504 153 504 145Q504 144 502 134Q486 77 440 33T333 -11Q263 -11 227 52Q186 -10 133 -10H127Q78 -10 57 16T35 71Q35 103 54 123T99 143Q142 143 142 101Q142 81 130 66T107 46T94 41L91 40Q91 39 97 36T113 29T132 26Q168 26 194 71Q203 87 217 139T245 247T261 313Q266 340 266 352Q266 380 251 392T217 404Q177 404 142 372T93 290Q91 281 88 280T72 278H58Q52 284 52 289Z"></path><path stroke-width="0" id="E280-MJMATHI-54" d="M40 437Q21 437 21 445Q21 450 37 501T71 602L88 651Q93 669 101 677H569H659Q691 677 697 676T704 667Q704 661 687 553T668 444Q668 437 649 437Q640 437 637 437T631 442L629 445Q629 451 635 490T641 551Q641 586 628 604T573 629Q568 630 515 631Q469 631 457 630T439 622Q438 621 368 343T298 60Q298 48 386 46Q418 46 427 45T436 36Q436 31 433 22Q429 4 424 1L422 0Q419 0 415 0Q410 0 363 1T228 2Q99 2 64 0H49Q43 6 43 9T45 27Q49 40 55 46H83H94Q174 46 189 55Q190 56 191 56Q196 59 201 76T241 233Q258 301 269 344Q339 619 339 625Q339 630 310 630H279Q212 630 191 624Q146 614 121 583T67 467Q60 445 57 441T43 437H40Z"></path><path stroke-width="0" id="E280-MJSZ4-28" d="M758 -1237T758 -1240T752 -1249H736Q718 -1249 717 -1248Q711 -1245 672 -1199Q237 -706 237 251T672 1700Q697 1730 716 1749Q718 1750 735 1750H752Q758 1744 758 1741Q758 1737 740 1713T689 1644T619 1537T540 1380T463 1176Q348 802 348 251Q348 -242 441 -599T744 -1218Q758 -1237 758 -1240Z"></path><path stroke-width="0" id="E280-MJSZ4-29" d="M33 1741Q33 1750 51 1750H60H65Q73 1750 81 1743T119 1700Q554 1207 554 251Q554 -707 119 -1199Q76 -1250 66 -1250Q65 -1250 62 -1250T56 -1249Q55 -1249 53 -1249T49 -1250Q33 -1250 33 -1239Q33 -1236 50 -1214T98 -1150T163 -1052T238 -910T311 -727Q443 -335 443 251Q443 402 436 532T405 831T339 1142T224 1438T50 1716Q33 1737 33 1741Z"></path><path stroke-width="0" id="E280-MJMAIN-2212" d="M84 237T84 250T98 270H679Q694 262 694 250T679 230H98Q84 237 84 250Z"></path><path stroke-width="0" id="E280-MJMAIN-31" d="M213 578L200 573Q186 568 160 563T102 556H83V602H102Q149 604 189 617T245 641T273 663Q275 666 285 666Q294 666 302 660V361L303 61Q310 54 315 52T339 48T401 46H427V0H416Q395 3 257 3Q121 3 100 0H88V46H114Q136 46 152 46T177 47T193 50T201 52T207 57T213 61V578Z"></path><path stroke-width="0" id="E280-MJMATHI-47" d="M50 252Q50 367 117 473T286 641T490 704Q580 704 633 653Q642 643 648 636T656 626L657 623Q660 623 684 649Q691 655 699 663T715 679T725 690L740 705H746Q760 705 760 698Q760 694 728 561Q692 422 692 421Q690 416 687 415T669 413H653Q647 419 647 422Q647 423 648 429T650 449T651 481Q651 552 619 605T510 659Q492 659 471 656T418 643T357 615T294 567T236 496T189 394T158 260Q156 242 156 221Q156 173 170 136T206 79T256 45T308 28T353 24Q407 24 452 47T514 106Q517 114 529 161T541 214Q541 222 528 224T468 227H431Q425 233 425 235T427 254Q431 267 437 273H454Q494 271 594 271Q634 271 659 271T695 272T707 272Q721 272 721 263Q721 261 719 249Q714 230 709 228Q706 227 694 227Q674 227 653 224Q646 221 643 215T629 164Q620 131 614 108Q589 6 586 3Q584 1 581 1Q571 1 553 21T530 52Q530 53 528 52T522 47Q448 -22 322 -22Q201 -22 126 55T50 252Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><g transform="translate(41043,0)"><g id="mjx-eqn-9" transform="translate(0,-99)"><use xlink:href="#E280-MJMAIN-28"></use><use xlink:href="#E280-MJMAIN-39" x="389" y="0"></use><use xlink:href="#E280-MJMAIN-29" x="889" y="0"></use></g></g><g transform="translate(9011,0)"><g transform="translate(-19,0)"><g transform="translate(0,-99)"><use xlink:href="#E280-MJMAINB-77" x="0" y="0"></use><g transform="translate(831,-155)"><use transform="scale(0.707)" xlink:href="#E280-MJMATHI-4C" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E280-MJMATHI-53" x="681" y="0"></use><use transform="scale(0.707)" xlink:href="#E280-MJMATHI-4D" x="1326" y="0"></use><use transform="scale(0.707)" xlink:href="#E280-MJMATHI-43" x="2377" y="0"></use></g><use xlink:href="#E280-MJMAIN-3D" x="3426" y="0"></use><g transform="translate(4482,0)"><use xlink:href="#E280-MJSZ4-28"></use><g transform="translate(792,0)"><use xlink:href="#E280-MJSZ2-2211" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E280-MJMATHI-74" x="840" y="-1499"></use></g><use xlink:href="#E280-MJMAINB-78" x="2402" y="0"></use><use xlink:href="#E280-MJMAIN-28" x="3009" y="0"></use><g transform="translate(3398,0)"><use xlink:href="#E280-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E280-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E280-MJMAIN-2C" x="4366" y="0"></use><g transform="translate(4811,0)"><use xlink:href="#E280-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E280-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E280-MJMAIN-29" x="5916" y="0"></use><use xlink:href="#E280-MJMAIN-28" x="6305" y="0"></use><use xlink:href="#E280-MJMATHI-78" x="6694" y="0"></use><use xlink:href="#E280-MJMAIN-28" x="7266" y="0"></use><g transform="translate(7655,0)"><use xlink:href="#E280-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E280-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E280-MJMAIN-2C" x="8624" y="0"></use><g transform="translate(9068,0)"><use xlink:href="#E280-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E280-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E280-MJMAIN-29" x="10174" y="0"></use><g transform="translate(10563,0)"><use xlink:href="#E280-MJMAIN-29" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E280-MJMATHI-54" x="550" y="583"></use></g><use xlink:href="#E280-MJSZ4-29" x="11549" y="0"></use><g transform="translate(12341,1476)"><use transform="scale(0.707)" xlink:href="#E280-MJMAIN-2212" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E280-MJMAIN-31" x="778" y="0"></use></g></g><g transform="translate(17994,0)"><use xlink:href="#E280-MJSZ2-2211" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E280-MJMATHI-74" x="840" y="-1499"></use></g><g transform="translate(19605,0)"><use xlink:href="#E280-MJMATHI-47" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E280-MJMATHI-74" x="1111" y="-213"></use></g><use xlink:href="#E280-MJMAINB-78" x="20746" y="0"></use><use xlink:href="#E280-MJMAIN-28" x="21353" y="0"></use><g transform="translate(21742,0)"><use xlink:href="#E280-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E280-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E280-MJMAIN-2C" x="22711" y="0"></use><g transform="translate(23155,0)"><use xlink:href="#E280-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E280-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E280-MJMAIN-29" x="24261" y="0"></use></g></g></g></g></svg></span></div><script type="math/tex; mode=display" id="MathJax-Element-253">\bold w_{LSMC} = \left(\sum_t \bold x(S_t,A_t)(x(S_t,A_t))^T \right)^{-1} \sum_t G_t\bold x(S_t,A_t)</script></div></div><p><span>直接使用上式更新权重，就实现了线性最小二乘回合更新。</span></p><p><span>将线性最小二乘用于时序差分，可以有</span><strong><span>线性最小二乘时序差分更新</span></strong><span>（Linear Least Square Temporal Difference, Linear LSTD）。对于单步时序差分，它试图最小化 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="31.052ex" height="5.218ex" viewBox="0 -998.8 13369.7 2246.8" role="img" focusable="false" style="vertical-align: -2.899ex;"><defs><path stroke-width="0" id="E313-MJMATHI-4C" d="M228 637Q194 637 192 641Q191 643 191 649Q191 673 202 682Q204 683 217 683Q271 680 344 680Q485 680 506 683H518Q524 677 524 674T522 656Q517 641 513 637H475Q406 636 394 628Q387 624 380 600T313 336Q297 271 279 198T252 88L243 52Q243 48 252 48T311 46H328Q360 46 379 47T428 54T478 72T522 106T564 161Q580 191 594 228T611 270Q616 273 628 273H641Q647 264 647 262T627 203T583 83T557 9Q555 4 553 3T537 0T494 -1Q483 -1 418 -1T294 0H116Q32 0 32 10Q32 17 34 24Q39 43 44 45Q48 46 59 46H65Q92 46 125 49Q139 52 144 61Q147 65 216 339T285 628Q285 635 228 637Z"></path><path stroke-width="0" id="E313-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E313-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E313-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E313-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E313-MJSZ2-2211" d="M60 948Q63 950 665 950H1267L1325 815Q1384 677 1388 669H1348L1341 683Q1320 724 1285 761Q1235 809 1174 838T1033 881T882 898T699 902H574H543H251L259 891Q722 258 724 252Q725 250 724 246Q721 243 460 -56L196 -356Q196 -357 407 -357Q459 -357 548 -357T676 -358Q812 -358 896 -353T1063 -332T1204 -283T1307 -196Q1328 -170 1348 -124H1388Q1388 -125 1381 -145T1356 -210T1325 -294L1267 -449L666 -450Q64 -450 61 -448Q55 -446 55 -439Q55 -437 57 -433L590 177Q590 178 557 222T452 366T322 544L56 909L55 924Q55 945 60 948Z"></path><path stroke-width="0" id="E313-MJMATHI-74" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path><path stroke-width="0" id="E313-MJMAIN-5B" d="M118 -250V750H255V710H158V-210H255V-250H118Z"></path><path stroke-width="0" id="E313-MJMATHI-55" d="M107 637Q73 637 71 641Q70 643 70 649Q70 673 81 682Q83 683 98 683Q139 681 234 681Q268 681 297 681T342 682T362 682Q378 682 378 672Q378 670 376 658Q371 641 366 638H364Q362 638 359 638T352 638T343 637T334 637Q295 636 284 634T266 623Q265 621 238 518T184 302T154 169Q152 155 152 140Q152 86 183 55T269 24Q336 24 403 69T501 205L552 406Q599 598 599 606Q599 633 535 637Q511 637 511 648Q511 650 513 660Q517 676 519 679T529 683Q532 683 561 682T645 680Q696 680 723 681T752 682Q767 682 767 672Q767 650 759 642Q756 637 737 637Q666 633 648 597Q646 592 598 404Q557 235 548 205Q515 105 433 42T263 -22Q171 -22 116 34T60 167V183Q60 201 115 421Q164 622 164 628Q164 635 107 637Z"></path><path stroke-width="0" id="E313-MJMAIN-2212" d="M84 237T84 250T98 270H679Q694 262 694 250T679 230H98Q84 237 84 250Z"></path><path stroke-width="0" id="E313-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E313-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E313-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E313-MJMATHI-41" d="M208 74Q208 50 254 46Q272 46 272 35Q272 34 270 22Q267 8 264 4T251 0Q249 0 239 0T205 1T141 2Q70 2 50 0H42Q35 7 35 11Q37 38 48 46H62Q132 49 164 96Q170 102 345 401T523 704Q530 716 547 716H555H572Q578 707 578 706L606 383Q634 60 636 57Q641 46 701 46Q726 46 726 36Q726 34 723 22Q720 7 718 4T704 0Q701 0 690 0T651 1T578 2Q484 2 455 0H443Q437 6 437 9T439 27Q443 40 445 43L449 46H469Q523 49 533 63L521 213H283L249 155Q208 86 208 74ZM516 260Q516 271 504 416T490 562L463 519Q447 492 400 412L310 260L413 259Q516 259 516 260Z"></path><path stroke-width="0" id="E313-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E313-MJMAIN-5D" d="M22 710V750H159V-250H22V-210H119V710H22Z"></path><path stroke-width="0" id="E313-MJMAIN-32" d="M109 429Q82 429 66 447T50 491Q50 562 103 614T235 666Q326 666 387 610T449 465Q449 422 429 383T381 315T301 241Q265 210 201 149L142 93L218 92Q375 92 385 97Q392 99 409 186V189H449V186Q448 183 436 95T421 3V0H50V19V31Q50 38 56 46T86 81Q115 113 136 137Q145 147 170 174T204 211T233 244T261 278T284 308T305 340T320 369T333 401T340 431T343 464Q343 527 309 573T212 619Q179 619 154 602T119 569T109 550Q109 549 114 549Q132 549 151 535T170 489Q170 464 154 447T109 429Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E313-MJMATHI-4C" x="0" y="0"></use><use xlink:href="#E313-MJMAIN-28" x="681" y="0"></use><use xlink:href="#E313-MJMAINB-77" x="1070" y="0"></use><use xlink:href="#E313-MJMAIN-29" x="1901" y="0"></use><use xlink:href="#E313-MJMAIN-3D" x="2567" y="0"></use><g transform="translate(3623,0)"><use xlink:href="#E313-MJSZ2-2211" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E313-MJMATHI-74" x="840" y="-1499"></use></g><use xlink:href="#E313-MJMAIN-5B" x="5067" y="0"></use><g transform="translate(5345,0)"><use xlink:href="#E313-MJMATHI-55" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E313-MJMATHI-74" x="965" y="-213"></use></g><use xlink:href="#E313-MJMAIN-2212" x="6606" y="0"></use><use xlink:href="#E313-MJMATHI-71" x="7606" y="0"></use><use xlink:href="#E313-MJMAIN-28" x="8066" y="0"></use><g transform="translate(8455,0)"><use xlink:href="#E313-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E313-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E313-MJMAIN-2C" x="9423" y="0"></use><g transform="translate(9868,0)"><use xlink:href="#E313-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E313-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E313-MJMAIN-3B" x="10973" y="0"></use><use xlink:href="#E313-MJMAINB-77" x="11418" y="0"></use><use xlink:href="#E313-MJMAIN-29" x="12249" y="0"></use><g transform="translate(12638,0)"><use xlink:href="#E313-MJMAIN-5D" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E313-MJMAIN-32" x="393" y="583"></use></g></g></svg></span><script type="math/tex">\displaystyle L(\bold w) = \sum_t[U_t - q(S_t,A_t;\bold w)]^2</script><span> ，其中 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="30.181ex" height="2.71ex" viewBox="0 -832.7 12994.4 1166.9" role="img" focusable="false" style="vertical-align: -0.776ex;"><defs><path stroke-width="0" id="E314-MJMATHI-55" d="M107 637Q73 637 71 641Q70 643 70 649Q70 673 81 682Q83 683 98 683Q139 681 234 681Q268 681 297 681T342 682T362 682Q378 682 378 672Q378 670 376 658Q371 641 366 638H364Q362 638 359 638T352 638T343 637T334 637Q295 636 284 634T266 623Q265 621 238 518T184 302T154 169Q152 155 152 140Q152 86 183 55T269 24Q336 24 403 69T501 205L552 406Q599 598 599 606Q599 633 535 637Q511 637 511 648Q511 650 513 660Q517 676 519 679T529 683Q532 683 561 682T645 680Q696 680 723 681T752 682Q767 682 767 672Q767 650 759 642Q756 637 737 637Q666 633 648 597Q646 592 598 404Q557 235 548 205Q515 105 433 42T263 -22Q171 -22 116 34T60 167V183Q60 201 115 421Q164 622 164 628Q164 635 107 637Z"></path><path stroke-width="0" id="E314-MJMATHI-74" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path><path stroke-width="0" id="E314-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E314-MJMATHI-52" d="M230 637Q203 637 198 638T193 649Q193 676 204 682Q206 683 378 683Q550 682 564 680Q620 672 658 652T712 606T733 563T739 529Q739 484 710 445T643 385T576 351T538 338L545 333Q612 295 612 223Q612 212 607 162T602 80V71Q602 53 603 43T614 25T640 16Q668 16 686 38T712 85Q717 99 720 102T735 105Q755 105 755 93Q755 75 731 36Q693 -21 641 -21H632Q571 -21 531 4T487 82Q487 109 502 166T517 239Q517 290 474 313Q459 320 449 321T378 323H309L277 193Q244 61 244 59Q244 55 245 54T252 50T269 48T302 46H333Q339 38 339 37T336 19Q332 6 326 0H311Q275 2 180 2Q146 2 117 2T71 2T50 1Q33 1 33 10Q33 12 36 24Q41 43 46 45Q50 46 61 46H67Q94 46 127 49Q141 52 146 61Q149 65 218 339T287 628Q287 635 230 637ZM630 554Q630 586 609 608T523 636Q521 636 500 636T462 637H440Q393 637 386 627Q385 624 352 494T319 361Q319 360 388 360Q466 361 492 367Q556 377 592 426Q608 449 619 486T630 554Z"></path><path stroke-width="0" id="E314-MJMAIN-2B" d="M56 237T56 250T70 270H369V420L370 570Q380 583 389 583Q402 583 409 568V270H707Q722 262 722 250T707 230H409V-68Q401 -82 391 -82H389H387Q375 -82 369 -68V230H70Q56 237 56 250Z"></path><path stroke-width="0" id="E314-MJMAIN-31" d="M213 578L200 573Q186 568 160 563T102 556H83V602H102Q149 604 189 617T245 641T273 663Q275 666 285 666Q294 666 302 660V361L303 61Q310 54 315 52T339 48T401 46H427V0H416Q395 3 257 3Q121 3 100 0H88V46H114Q136 46 152 46T177 47T193 50T201 52T207 57T213 61V578Z"></path><path stroke-width="0" id="E314-MJMATHI-3B3" d="M31 249Q11 249 11 258Q11 275 26 304T66 365T129 418T206 441Q233 441 239 440Q287 429 318 386T371 255Q385 195 385 170Q385 166 386 166L398 193Q418 244 443 300T486 391T508 430Q510 431 524 431H537Q543 425 543 422Q543 418 522 378T463 251T391 71Q385 55 378 6T357 -100Q341 -165 330 -190T303 -216Q286 -216 286 -188Q286 -138 340 32L346 51L347 69Q348 79 348 100Q348 257 291 317Q251 355 196 355Q148 355 108 329T51 260Q49 251 47 251Q45 249 31 249Z"></path><path stroke-width="0" id="E314-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E314-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E314-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E314-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E314-MJMATHI-41" d="M208 74Q208 50 254 46Q272 46 272 35Q272 34 270 22Q267 8 264 4T251 0Q249 0 239 0T205 1T141 2Q70 2 50 0H42Q35 7 35 11Q37 38 48 46H62Q132 49 164 96Q170 102 345 401T523 704Q530 716 547 716H555H572Q578 707 578 706L606 383Q634 60 636 57Q641 46 701 46Q726 46 726 36Q726 34 723 22Q720 7 718 4T704 0Q701 0 690 0T651 1T578 2Q484 2 455 0H443Q437 6 437 9T439 27Q443 40 445 43L449 46H469Q523 49 533 63L521 213H283L249 155Q208 86 208 74ZM516 260Q516 271 504 416T490 562L463 519Q447 492 400 412L310 260L413 259Q516 259 516 260Z"></path><path stroke-width="0" id="E314-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E314-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E314-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E314-MJMATHI-55" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E314-MJMATHI-74" x="965" y="-213"></use><use xlink:href="#E314-MJMAIN-3D" x="1316" y="0"></use><g transform="translate(2371,0)"><use xlink:href="#E314-MJMATHI-52" x="0" y="0"></use><g transform="translate(759,-150)"><use transform="scale(0.707)" xlink:href="#E314-MJMATHI-74" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E314-MJMAIN-2B" x="361" y="0"></use><use transform="scale(0.707)" xlink:href="#E314-MJMAIN-31" x="1139" y="0"></use></g></g><use xlink:href="#E314-MJMAIN-2B" x="4611" y="0"></use><use xlink:href="#E314-MJMATHI-3B3" x="5612" y="0"></use><use xlink:href="#E314-MJMATHI-71" x="6155" y="0"></use><use xlink:href="#E314-MJMAIN-28" x="6615" y="0"></use><g transform="translate(7004,0)"><use xlink:href="#E314-MJMATHI-53" x="0" y="0"></use><g transform="translate(613,-150)"><use transform="scale(0.707)" xlink:href="#E314-MJMATHI-74" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E314-MJMAIN-2B" x="361" y="0"></use><use transform="scale(0.707)" xlink:href="#E314-MJMAIN-31" x="1139" y="0"></use></g></g><use xlink:href="#E314-MJMAIN-2C" x="8876" y="0"></use><g transform="translate(9320,0)"><use xlink:href="#E314-MJMATHI-41" x="0" y="0"></use><g transform="translate(750,-150)"><use transform="scale(0.707)" xlink:href="#E314-MJMATHI-74" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E314-MJMAIN-2B" x="361" y="0"></use><use transform="scale(0.707)" xlink:href="#E314-MJMAIN-31" x="1139" y="0"></use></g></g><use xlink:href="#E314-MJMAIN-3B" x="11329" y="0"></use><use xlink:href="#E314-MJMAINB-77" x="11774" y="0"></use><use xlink:href="#E314-MJMAIN-29" x="12605" y="0"></use></g></svg></span><script type="math/tex">U_t = R_{t+1} + \gamma q(S_{t+1},A_{t+1};\bold w)</script><span> ；与回合更新类似，但这里是对 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="5.319ex" height="2.71ex" viewBox="0 -832.7 2290 1166.9" role="img" focusable="false" style="vertical-align: -0.776ex;"><defs><path stroke-width="0" id="E317-MJMATHI-4C" d="M228 637Q194 637 192 641Q191 643 191 649Q191 673 202 682Q204 683 217 683Q271 680 344 680Q485 680 506 683H518Q524 677 524 674T522 656Q517 641 513 637H475Q406 636 394 628Q387 624 380 600T313 336Q297 271 279 198T252 88L243 52Q243 48 252 48T311 46H328Q360 46 379 47T428 54T478 72T522 106T564 161Q580 191 594 228T611 270Q616 273 628 273H641Q647 264 647 262T627 203T583 83T557 9Q555 4 553 3T537 0T494 -1Q483 -1 418 -1T294 0H116Q32 0 32 10Q32 17 34 24Q39 43 44 45Q48 46 59 46H65Q92 46 125 49Q139 52 144 61Q147 65 216 339T285 628Q285 635 228 637Z"></path><path stroke-width="0" id="E317-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E317-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E317-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E317-MJMATHI-4C" x="0" y="0"></use><use xlink:href="#E317-MJMAIN-28" x="681" y="0"></use><use xlink:href="#E317-MJMAINB-77" x="1070" y="0"></use><use xlink:href="#E317-MJMAIN-29" x="1901" y="0"></use></g></svg></span><script type="math/tex">L(\bold w)</script><span> 求半梯度，最后可以求解得到：</span></p><div contenteditable="false" spellcheck="false" class="mathjax-block md-end-block md-math-block md-rawblock" id="mathjax-n43" cid="n43" mdtype="math_block"><div class="md-rawblock-container md-math-container" tabindex="-1"><div class="MathJax_SVG_Display"><span class="MathJax_SVG" id="MathJax-Element-254-Frame" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="98.296ex" height="7.726ex" viewBox="0 -1912.5 42321.7 3326.6" role="img" focusable="false" style="vertical-align: -3.284ex; max-width: 100%;"><defs><path stroke-width="0" id="E281-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E281-MJMAIN-31" d="M213 578L200 573Q186 568 160 563T102 556H83V602H102Q149 604 189 617T245 641T273 663Q275 666 285 666Q294 666 302 660V361L303 61Q310 54 315 52T339 48T401 46H427V0H416Q395 3 257 3Q121 3 100 0H88V46H114Q136 46 152 46T177 47T193 50T201 52T207 57T213 61V578Z"></path><path stroke-width="0" id="E281-MJMAIN-30" d="M96 585Q152 666 249 666Q297 666 345 640T423 548Q460 465 460 320Q460 165 417 83Q397 41 362 16T301 -15T250 -22Q224 -22 198 -16T137 16T82 83Q39 165 39 320Q39 494 96 585ZM321 597Q291 629 250 629Q208 629 178 597Q153 571 145 525T137 333Q137 175 145 125T181 46Q209 16 250 16Q290 16 318 46Q347 76 354 130T362 333Q362 478 354 524T321 597Z"></path><path stroke-width="0" id="E281-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E281-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E281-MJMATHI-4C" d="M228 637Q194 637 192 641Q191 643 191 649Q191 673 202 682Q204 683 217 683Q271 680 344 680Q485 680 506 683H518Q524 677 524 674T522 656Q517 641 513 637H475Q406 636 394 628Q387 624 380 600T313 336Q297 271 279 198T252 88L243 52Q243 48 252 48T311 46H328Q360 46 379 47T428 54T478 72T522 106T564 161Q580 191 594 228T611 270Q616 273 628 273H641Q647 264 647 262T627 203T583 83T557 9Q555 4 553 3T537 0T494 -1Q483 -1 418 -1T294 0H116Q32 0 32 10Q32 17 34 24Q39 43 44 45Q48 46 59 46H65Q92 46 125 49Q139 52 144 61Q147 65 216 339T285 628Q285 635 228 637Z"></path><path stroke-width="0" id="E281-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E281-MJMATHI-54" d="M40 437Q21 437 21 445Q21 450 37 501T71 602L88 651Q93 669 101 677H569H659Q691 677 697 676T704 667Q704 661 687 553T668 444Q668 437 649 437Q640 437 637 437T631 442L629 445Q629 451 635 490T641 551Q641 586 628 604T573 629Q568 630 515 631Q469 631 457 630T439 622Q438 621 368 343T298 60Q298 48 386 46Q418 46 427 45T436 36Q436 31 433 22Q429 4 424 1L422 0Q419 0 415 0Q410 0 363 1T228 2Q99 2 64 0H49Q43 6 43 9T45 27Q49 40 55 46H83H94Q174 46 189 55Q190 56 191 56Q196 59 201 76T241 233Q258 301 269 344Q339 619 339 625Q339 630 310 630H279Q212 630 191 624Q146 614 121 583T67 467Q60 445 57 441T43 437H40Z"></path><path stroke-width="0" id="E281-MJMATHI-44" d="M287 628Q287 635 230 637Q207 637 200 638T193 647Q193 655 197 667T204 682Q206 683 403 683Q570 682 590 682T630 676Q702 659 752 597T803 431Q803 275 696 151T444 3L430 1L236 0H125H72Q48 0 41 2T33 11Q33 13 36 25Q40 41 44 43T67 46Q94 46 127 49Q141 52 146 61Q149 65 218 339T287 628ZM703 469Q703 507 692 537T666 584T629 613T590 629T555 636Q553 636 541 636T512 636T479 637H436Q392 637 386 627Q384 623 313 339T242 52Q242 48 253 48T330 47Q335 47 349 47T373 46Q499 46 581 128Q617 164 640 212T683 339T703 469Z"></path><path stroke-width="0" id="E281-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E281-MJSZ2-2211" d="M60 948Q63 950 665 950H1267L1325 815Q1384 677 1388 669H1348L1341 683Q1320 724 1285 761Q1235 809 1174 838T1033 881T882 898T699 902H574H543H251L259 891Q722 258 724 252Q725 250 724 246Q721 243 460 -56L196 -356Q196 -357 407 -357Q459 -357 548 -357T676 -358Q812 -358 896 -353T1063 -332T1204 -283T1307 -196Q1328 -170 1348 -124H1388Q1388 -125 1381 -145T1356 -210T1325 -294L1267 -449L666 -450Q64 -450 61 -448Q55 -446 55 -439Q55 -437 57 -433L590 177Q590 178 557 222T452 366T322 544L56 909L55 924Q55 945 60 948Z"></path><path stroke-width="0" id="E281-MJMATHI-74" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path><path stroke-width="0" id="E281-MJMAINB-78" d="M227 0Q212 3 121 3Q40 3 28 0H21V62H117L245 213L109 382H26V444H34Q49 441 143 441Q247 441 265 444H274V382H246L281 339Q315 297 316 297Q320 297 354 341L389 382H352V444H360Q375 441 466 441Q547 441 559 444H566V382H471L355 246L504 63L545 62H586V0H578Q563 3 469 3Q365 3 347 0H338V62H366Q366 63 326 112T285 163L198 63L217 62H235V0H227Z"></path><path stroke-width="0" id="E281-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E281-MJMATHI-41" d="M208 74Q208 50 254 46Q272 46 272 35Q272 34 270 22Q267 8 264 4T251 0Q249 0 239 0T205 1T141 2Q70 2 50 0H42Q35 7 35 11Q37 38 48 46H62Q132 49 164 96Q170 102 345 401T523 704Q530 716 547 716H555H572Q578 707 578 706L606 383Q634 60 636 57Q641 46 701 46Q726 46 726 36Q726 34 723 22Q720 7 718 4T704 0Q701 0 690 0T651 1T578 2Q484 2 455 0H443Q437 6 437 9T439 27Q443 40 445 43L449 46H469Q523 49 533 63L521 213H283L249 155Q208 86 208 74ZM516 260Q516 271 504 416T490 562L463 519Q447 492 400 412L310 260L413 259Q516 259 516 260Z"></path><path stroke-width="0" id="E281-MJMATHI-78" d="M52 289Q59 331 106 386T222 442Q257 442 286 424T329 379Q371 442 430 442Q467 442 494 420T522 361Q522 332 508 314T481 292T458 288Q439 288 427 299T415 328Q415 374 465 391Q454 404 425 404Q412 404 406 402Q368 386 350 336Q290 115 290 78Q290 50 306 38T341 26Q378 26 414 59T463 140Q466 150 469 151T485 153H489Q504 153 504 145Q504 144 502 134Q486 77 440 33T333 -11Q263 -11 227 52Q186 -10 133 -10H127Q78 -10 57 16T35 71Q35 103 54 123T99 143Q142 143 142 101Q142 81 130 66T107 46T94 41L91 40Q91 39 97 36T113 29T132 26Q168 26 194 71Q203 87 217 139T245 247T261 313Q266 340 266 352Q266 380 251 392T217 404Q177 404 142 372T93 290Q91 281 88 280T72 278H58Q52 284 52 289Z"></path><path stroke-width="0" id="E281-MJMAIN-2212" d="M84 237T84 250T98 270H679Q694 262 694 250T679 230H98Q84 237 84 250Z"></path><path stroke-width="0" id="E281-MJMATHI-3B3" d="M31 249Q11 249 11 258Q11 275 26 304T66 365T129 418T206 441Q233 441 239 440Q287 429 318 386T371 255Q385 195 385 170Q385 166 386 166L398 193Q418 244 443 300T486 391T508 430Q510 431 524 431H537Q543 425 543 422Q543 418 522 378T463 251T391 71Q385 55 378 6T357 -100Q341 -165 330 -190T303 -216Q286 -216 286 -188Q286 -138 340 32L346 51L347 69Q348 79 348 100Q348 257 291 317Q251 355 196 355Q148 355 108 329T51 260Q49 251 47 251Q45 249 31 249Z"></path><path stroke-width="0" id="E281-MJMAIN-2B" d="M56 237T56 250T70 270H369V420L370 570Q380 583 389 583Q402 583 409 568V270H707Q722 262 722 250T707 230H409V-68Q401 -82 391 -82H389H387Q375 -82 369 -68V230H70Q56 237 56 250Z"></path><path stroke-width="0" id="E281-MJSZ4-28" d="M758 -1237T758 -1240T752 -1249H736Q718 -1249 717 -1248Q711 -1245 672 -1199Q237 -706 237 251T672 1700Q697 1730 716 1749Q718 1750 735 1750H752Q758 1744 758 1741Q758 1737 740 1713T689 1644T619 1537T540 1380T463 1176Q348 802 348 251Q348 -242 441 -599T744 -1218Q758 -1237 758 -1240Z"></path><path stroke-width="0" id="E281-MJSZ4-29" d="M33 1741Q33 1750 51 1750H60H65Q73 1750 81 1743T119 1700Q554 1207 554 251Q554 -707 119 -1199Q76 -1250 66 -1250Q65 -1250 62 -1250T56 -1249Q55 -1249 53 -1249T49 -1250Q33 -1250 33 -1239Q33 -1236 50 -1214T98 -1150T163 -1052T238 -910T311 -727Q443 -335 443 251Q443 402 436 532T405 831T339 1142T224 1438T50 1716Q33 1737 33 1741Z"></path><path stroke-width="0" id="E281-MJMATHI-52" d="M230 637Q203 637 198 638T193 649Q193 676 204 682Q206 683 378 683Q550 682 564 680Q620 672 658 652T712 606T733 563T739 529Q739 484 710 445T643 385T576 351T538 338L545 333Q612 295 612 223Q612 212 607 162T602 80V71Q602 53 603 43T614 25T640 16Q668 16 686 38T712 85Q717 99 720 102T735 105Q755 105 755 93Q755 75 731 36Q693 -21 641 -21H632Q571 -21 531 4T487 82Q487 109 502 166T517 239Q517 290 474 313Q459 320 449 321T378 323H309L277 193Q244 61 244 59Q244 55 245 54T252 50T269 48T302 46H333Q339 38 339 37T336 19Q332 6 326 0H311Q275 2 180 2Q146 2 117 2T71 2T50 1Q33 1 33 10Q33 12 36 24Q41 43 46 45Q50 46 61 46H67Q94 46 127 49Q141 52 146 61Q149 65 218 339T287 628Q287 635 230 637ZM630 554Q630 586 609 608T523 636Q521 636 500 636T462 637H440Q393 637 386 627Q385 624 352 494T319 361Q319 360 388 360Q466 361 492 367Q556 377 592 426Q608 449 619 486T630 554Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><g transform="translate(40543,0)"><g id="mjx-eqn-eq:10_1" transform="translate(0,-99)"><use xlink:href="#E281-MJMAIN-28"></use><use xlink:href="#E281-MJMAIN-31" x="389" y="0"></use><use xlink:href="#E281-MJMAIN-30" x="889" y="0"></use><use xlink:href="#E281-MJMAIN-29" x="1389" y="0"></use></g></g><g transform="translate(4934,0)"><g transform="translate(-19,0)"><g transform="translate(0,-99)"><use xlink:href="#E281-MJMAINB-77" x="0" y="0"></use><g transform="translate(831,-155)"><use transform="scale(0.707)" xlink:href="#E281-MJMATHI-4C" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E281-MJMATHI-53" x="681" y="0"></use><use transform="scale(0.707)" xlink:href="#E281-MJMATHI-54" x="1326" y="0"></use><use transform="scale(0.707)" xlink:href="#E281-MJMATHI-44" x="2029" y="0"></use></g><use xlink:href="#E281-MJMAIN-3D" x="3229" y="0"></use><g transform="translate(4285,0)"><use xlink:href="#E281-MJSZ4-28"></use><g transform="translate(792,0)"><use xlink:href="#E281-MJSZ2-2211" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E281-MJMATHI-74" x="840" y="-1499"></use></g><use xlink:href="#E281-MJMAINB-78" x="2402" y="0"></use><use xlink:href="#E281-MJMAIN-28" x="3009" y="0"></use><g transform="translate(3398,0)"><use xlink:href="#E281-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E281-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E281-MJMAIN-2C" x="4366" y="0"></use><g transform="translate(4811,0)"><use xlink:href="#E281-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E281-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E281-MJMAIN-29" x="5916" y="0"></use><use xlink:href="#E281-MJMAIN-28" x="6305" y="0"></use><use xlink:href="#E281-MJMATHI-78" x="6694" y="0"></use><use xlink:href="#E281-MJMAIN-28" x="7266" y="0"></use><g transform="translate(7655,0)"><use xlink:href="#E281-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E281-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E281-MJMAIN-2C" x="8624" y="0"></use><g transform="translate(9068,0)"><use xlink:href="#E281-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E281-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E281-MJMAIN-29" x="10174" y="0"></use><use xlink:href="#E281-MJMAIN-2212" x="10785" y="0"></use><use xlink:href="#E281-MJMATHI-3B3" x="11785" y="0"></use><use xlink:href="#E281-MJMAINB-78" x="12328" y="0"></use><use xlink:href="#E281-MJMAIN-28" x="12935" y="0"></use><g transform="translate(13324,0)"><use xlink:href="#E281-MJMATHI-53" x="0" y="0"></use><g transform="translate(613,-150)"><use transform="scale(0.707)" xlink:href="#E281-MJMATHI-74" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E281-MJMAIN-2B" x="361" y="0"></use><use transform="scale(0.707)" xlink:href="#E281-MJMAIN-31" x="1139" y="0"></use></g></g><use xlink:href="#E281-MJMAIN-2C" x="15196" y="0"></use><g transform="translate(15641,0)"><use xlink:href="#E281-MJMATHI-41" x="0" y="0"></use><g transform="translate(750,-150)"><use transform="scale(0.707)" xlink:href="#E281-MJMATHI-74" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E281-MJMAIN-2B" x="361" y="0"></use><use transform="scale(0.707)" xlink:href="#E281-MJMAIN-31" x="1139" y="0"></use></g></g><use xlink:href="#E281-MJMAIN-29" x="17650" y="0"></use><g transform="translate(18039,0)"><use xlink:href="#E281-MJMAIN-29" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E281-MJMATHI-54" x="550" y="583"></use></g><use xlink:href="#E281-MJSZ4-29" x="19025" y="0"></use><g transform="translate(19817,1476)"><use transform="scale(0.707)" xlink:href="#E281-MJMAIN-2212" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E281-MJMAIN-31" x="778" y="0"></use></g></g><g transform="translate(25273,0)"><use xlink:href="#E281-MJSZ2-2211" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E281-MJMATHI-74" x="840" y="-1499"></use></g><g transform="translate(26884,0)"><use xlink:href="#E281-MJMATHI-52" x="0" y="0"></use><g transform="translate(759,-150)"><use transform="scale(0.707)" xlink:href="#E281-MJMATHI-74" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E281-MJMAIN-2B" x="361" y="0"></use><use transform="scale(0.707)" xlink:href="#E281-MJMAIN-31" x="1139" y="0"></use></g></g><use xlink:href="#E281-MJMAINB-78" x="28902" y="0"></use><use xlink:href="#E281-MJMAIN-28" x="29509" y="0"></use><g transform="translate(29898,0)"><use xlink:href="#E281-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E281-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E281-MJMAIN-2C" x="30866" y="0"></use><g transform="translate(31311,0)"><use xlink:href="#E281-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E281-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E281-MJMAIN-29" x="32416" y="0"></use></g></g></g></g></svg></span></div><script type="math/tex; mode=display" id="MathJax-Element-254">\bold w_{LSTD} = \left(\sum_t \bold x(S_t,A_t)(x(S_t,A_t) - \gamma\bold x(S_{t+1},A_{t+1}))^T \right)^{-1} \sum_t R_{t+1}\bold x(S_t,A_t)
\label{eq:10}</script></div></div><p><span>最小二乘也能用于最优策略求解。相对于单步时序差分，Q 学习只修改了回报估计为 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="35.263ex" height="4.35ex" viewBox="0 -832.7 15182.5 1873" role="img" focusable="false" style="vertical-align: -2.416ex;"><defs><path stroke-width="0" id="E316-MJMATHI-55" d="M107 637Q73 637 71 641Q70 643 70 649Q70 673 81 682Q83 683 98 683Q139 681 234 681Q268 681 297 681T342 682T362 682Q378 682 378 672Q378 670 376 658Q371 641 366 638H364Q362 638 359 638T352 638T343 637T334 637Q295 636 284 634T266 623Q265 621 238 518T184 302T154 169Q152 155 152 140Q152 86 183 55T269 24Q336 24 403 69T501 205L552 406Q599 598 599 606Q599 633 535 637Q511 637 511 648Q511 650 513 660Q517 676 519 679T529 683Q532 683 561 682T645 680Q696 680 723 681T752 682Q767 682 767 672Q767 650 759 642Q756 637 737 637Q666 633 648 597Q646 592 598 404Q557 235 548 205Q515 105 433 42T263 -22Q171 -22 116 34T60 167V183Q60 201 115 421Q164 622 164 628Q164 635 107 637Z"></path><path stroke-width="0" id="E316-MJMATHI-74" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path><path stroke-width="0" id="E316-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E316-MJMATHI-52" d="M230 637Q203 637 198 638T193 649Q193 676 204 682Q206 683 378 683Q550 682 564 680Q620 672 658 652T712 606T733 563T739 529Q739 484 710 445T643 385T576 351T538 338L545 333Q612 295 612 223Q612 212 607 162T602 80V71Q602 53 603 43T614 25T640 16Q668 16 686 38T712 85Q717 99 720 102T735 105Q755 105 755 93Q755 75 731 36Q693 -21 641 -21H632Q571 -21 531 4T487 82Q487 109 502 166T517 239Q517 290 474 313Q459 320 449 321T378 323H309L277 193Q244 61 244 59Q244 55 245 54T252 50T269 48T302 46H333Q339 38 339 37T336 19Q332 6 326 0H311Q275 2 180 2Q146 2 117 2T71 2T50 1Q33 1 33 10Q33 12 36 24Q41 43 46 45Q50 46 61 46H67Q94 46 127 49Q141 52 146 61Q149 65 218 339T287 628Q287 635 230 637ZM630 554Q630 586 609 608T523 636Q521 636 500 636T462 637H440Q393 637 386 627Q385 624 352 494T319 361Q319 360 388 360Q466 361 492 367Q556 377 592 426Q608 449 619 486T630 554Z"></path><path stroke-width="0" id="E316-MJMAIN-2B" d="M56 237T56 250T70 270H369V420L370 570Q380 583 389 583Q402 583 409 568V270H707Q722 262 722 250T707 230H409V-68Q401 -82 391 -82H389H387Q375 -82 369 -68V230H70Q56 237 56 250Z"></path><path stroke-width="0" id="E316-MJMAIN-31" d="M213 578L200 573Q186 568 160 563T102 556H83V602H102Q149 604 189 617T245 641T273 663Q275 666 285 666Q294 666 302 660V361L303 61Q310 54 315 52T339 48T401 46H427V0H416Q395 3 257 3Q121 3 100 0H88V46H114Q136 46 152 46T177 47T193 50T201 52T207 57T213 61V578Z"></path><path stroke-width="0" id="E316-MJMATHI-3B3" d="M31 249Q11 249 11 258Q11 275 26 304T66 365T129 418T206 441Q233 441 239 440Q287 429 318 386T371 255Q385 195 385 170Q385 166 386 166L398 193Q418 244 443 300T486 391T508 430Q510 431 524 431H537Q543 425 543 422Q543 418 522 378T463 251T391 71Q385 55 378 6T357 -100Q341 -165 330 -190T303 -216Q286 -216 286 -188Q286 -138 340 32L346 51L347 69Q348 79 348 100Q348 257 291 317Q251 355 196 355Q148 355 108 329T51 260Q49 251 47 251Q45 249 31 249Z"></path><path stroke-width="0" id="E316-MJMAIN-6D" d="M41 46H55Q94 46 102 60V68Q102 77 102 91T102 122T103 161T103 203Q103 234 103 269T102 328V351Q99 370 88 376T43 385H25V408Q25 431 27 431L37 432Q47 433 65 434T102 436Q119 437 138 438T167 441T178 442H181V402Q181 364 182 364T187 369T199 384T218 402T247 421T285 437Q305 442 336 442Q351 442 364 440T387 434T406 426T421 417T432 406T441 395T448 384T452 374T455 366L457 361L460 365Q463 369 466 373T475 384T488 397T503 410T523 422T546 432T572 439T603 442Q729 442 740 329Q741 322 741 190V104Q741 66 743 59T754 49Q775 46 803 46H819V0H811L788 1Q764 2 737 2T699 3Q596 3 587 0H579V46H595Q656 46 656 62Q657 64 657 200Q656 335 655 343Q649 371 635 385T611 402T585 404Q540 404 506 370Q479 343 472 315T464 232V168V108Q464 78 465 68T468 55T477 49Q498 46 526 46H542V0H534L510 1Q487 2 460 2T422 3Q319 3 310 0H302V46H318Q379 46 379 62Q380 64 380 200Q379 335 378 343Q372 371 358 385T334 402T308 404Q263 404 229 370Q202 343 195 315T187 232V168V108Q187 78 188 68T191 55T200 49Q221 46 249 46H265V0H257L234 1Q210 2 183 2T145 3Q42 3 33 0H25V46H41Z"></path><path stroke-width="0" id="E316-MJMAIN-61" d="M137 305T115 305T78 320T63 359Q63 394 97 421T218 448Q291 448 336 416T396 340Q401 326 401 309T402 194V124Q402 76 407 58T428 40Q443 40 448 56T453 109V145H493V106Q492 66 490 59Q481 29 455 12T400 -6T353 12T329 54V58L327 55Q325 52 322 49T314 40T302 29T287 17T269 6T247 -2T221 -8T190 -11Q130 -11 82 20T34 107Q34 128 41 147T68 188T116 225T194 253T304 268H318V290Q318 324 312 340Q290 411 215 411Q197 411 181 410T156 406T148 403Q170 388 170 359Q170 334 154 320ZM126 106Q126 75 150 51T209 26Q247 26 276 49T315 109Q317 116 318 175Q318 233 317 233Q309 233 296 232T251 223T193 203T147 166T126 106Z"></path><path stroke-width="0" id="E316-MJMAIN-78" d="M201 0Q189 3 102 3Q26 3 17 0H11V46H25Q48 47 67 52T96 61T121 78T139 96T160 122T180 150L226 210L168 288Q159 301 149 315T133 336T122 351T113 363T107 370T100 376T94 379T88 381T80 383Q74 383 44 385H16V431H23Q59 429 126 429Q219 429 229 431H237V385Q201 381 201 369Q201 367 211 353T239 315T268 274L272 270L297 304Q329 345 329 358Q329 364 327 369T322 376T317 380T310 384L307 385H302V431H309Q324 428 408 428Q487 428 493 431H499V385H492Q443 385 411 368Q394 360 377 341T312 257L296 236L358 151Q424 61 429 57T446 50Q464 46 499 46H516V0H510H502Q494 1 482 1T457 2T432 2T414 3Q403 3 377 3T327 1L304 0H295V46H298Q309 46 320 51T331 63Q331 65 291 120L250 175Q249 174 219 133T185 88Q181 83 181 74Q181 63 188 55T206 46Q208 46 208 23V0H201Z"></path><path stroke-width="0" id="E316-MJMATHI-61" d="M33 157Q33 258 109 349T280 441Q331 441 370 392Q386 422 416 422Q429 422 439 414T449 394Q449 381 412 234T374 68Q374 43 381 35T402 26Q411 27 422 35Q443 55 463 131Q469 151 473 152Q475 153 483 153H487Q506 153 506 144Q506 138 501 117T481 63T449 13Q436 0 417 -8Q409 -10 393 -10Q359 -10 336 5T306 36L300 51Q299 52 296 50Q294 48 292 46Q233 -10 172 -10Q117 -10 75 30T33 157ZM351 328Q351 334 346 350T323 385T277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q217 26 254 59T298 110Q300 114 325 217T351 328Z"></path><path stroke-width="0" id="E316-MJMAIN-2208" d="M84 250Q84 372 166 450T360 539Q361 539 377 539T419 540T469 540H568Q583 532 583 520Q583 511 570 501L466 500Q355 499 329 494Q280 482 242 458T183 409T147 354T129 306T124 272V270H568Q583 262 583 250T568 230H124V228Q124 207 134 177T167 112T231 48T328 7Q355 1 466 0H570Q583 -10 583 -20Q583 -32 568 -40H471Q464 -40 446 -40T417 -41Q262 -41 172 45Q84 127 84 250Z"></path><path stroke-width="0" id="E316-MJCAL-41" d="M576 668Q576 688 606 708T660 728Q676 728 675 712V571Q675 409 688 252Q696 122 720 57Q722 53 723 50T728 46T732 43T737 41T743 39L754 45Q788 61 803 61Q819 61 819 47Q818 43 814 35Q799 15 755 -7T675 -30Q659 -30 648 -25T630 -8T621 11T614 34Q603 77 599 106T594 146T591 160V163H460L329 164L316 145Q241 35 196 -7T119 -50T59 -24T30 43Q30 75 46 100T74 125Q81 125 83 120T88 104T96 84Q118 57 151 57Q189 57 277 182Q432 400 542 625L559 659H567Q574 659 575 660T576 668ZM584 249Q579 333 577 386T575 473T574 520V581L563 560Q497 426 412 290L372 228L370 224H371L383 228L393 232H586L584 249Z"></path><path stroke-width="0" id="E316-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E316-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E316-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E316-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E316-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E316-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E316-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E316-MJMATHI-55" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E316-MJMATHI-74" x="965" y="-213"></use><use xlink:href="#E316-MJMAIN-3D" x="1316" y="0"></use><g transform="translate(2371,0)"><use xlink:href="#E316-MJMATHI-52" x="0" y="0"></use><g transform="translate(759,-150)"><use transform="scale(0.707)" xlink:href="#E316-MJMATHI-74" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E316-MJMAIN-2B" x="361" y="0"></use><use transform="scale(0.707)" xlink:href="#E316-MJMAIN-31" x="1139" y="0"></use></g></g><use xlink:href="#E316-MJMAIN-2B" x="4611" y="0"></use><use xlink:href="#E316-MJMATHI-3B3" x="5612" y="0"></use><g transform="translate(6321,0)"><g transform="translate(736,0)"><use xlink:href="#E316-MJMAIN-6D"></use><use xlink:href="#E316-MJMAIN-61" x="833" y="0"></use><use xlink:href="#E316-MJMAIN-78" x="1333" y="0"></use></g><g transform="translate(0,-708)"><use transform="scale(0.707)" xlink:href="#E316-MJMATHI-61" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E316-MJMAIN-2208" x="529" y="0"></use><use transform="scale(0.707)" xlink:href="#E316-MJCAL-41" x="1196" y="0"></use><use transform="scale(0.707)" xlink:href="#E316-MJMAIN-28" x="2015" y="0"></use><use transform="scale(0.707)" xlink:href="#E316-MJMATHI-53" x="2404" y="0"></use><use transform="scale(0.707)" xlink:href="#E316-MJMAIN-2B" x="3049" y="0"></use><use transform="scale(0.707)" xlink:href="#E316-MJMAIN-31" x="3827" y="0"></use><use transform="scale(0.707)" xlink:href="#E316-MJMAIN-29" x="4327" y="0"></use></g></g><use xlink:href="#E316-MJMATHI-71" x="9823" y="0"></use><use xlink:href="#E316-MJMAIN-28" x="10283" y="0"></use><g transform="translate(10672,0)"><use xlink:href="#E316-MJMATHI-53" x="0" y="0"></use><g transform="translate(613,-150)"><use transform="scale(0.707)" xlink:href="#E316-MJMATHI-74" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E316-MJMAIN-2B" x="361" y="0"></use><use transform="scale(0.707)" xlink:href="#E316-MJMAIN-31" x="1139" y="0"></use></g></g><use xlink:href="#E316-MJMAIN-2C" x="12544" y="0"></use><use xlink:href="#E316-MJMATHI-61" x="12988" y="0"></use><use xlink:href="#E316-MJMAIN-3B" x="13517" y="0"></use><use xlink:href="#E316-MJMAINB-77" x="13962" y="0"></use><use xlink:href="#E316-MJMAIN-29" x="14793" y="0"></use></g></svg></span><script type="math/tex">\displaystyle U_t = R_{t+1} + \gamma \max_{a\in \mathcal A(S+1)} q(S_{t+1},a;\bold w)</script><span> ，并不影响对单步时序差分的最小化目标 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="5.319ex" height="2.71ex" viewBox="0 -832.7 2290 1166.9" role="img" focusable="false" style="vertical-align: -0.776ex;"><defs><path stroke-width="0" id="E317-MJMATHI-4C" d="M228 637Q194 637 192 641Q191 643 191 649Q191 673 202 682Q204 683 217 683Q271 680 344 680Q485 680 506 683H518Q524 677 524 674T522 656Q517 641 513 637H475Q406 636 394 628Q387 624 380 600T313 336Q297 271 279 198T252 88L243 52Q243 48 252 48T311 46H328Q360 46 379 47T428 54T478 72T522 106T564 161Q580 191 594 228T611 270Q616 273 628 273H641Q647 264 647 262T627 203T583 83T557 9Q555 4 553 3T537 0T494 -1Q483 -1 418 -1T294 0H116Q32 0 32 10Q32 17 34 24Q39 43 44 45Q48 46 59 46H65Q92 46 125 49Q139 52 144 61Q147 65 216 339T285 628Q285 635 228 637Z"></path><path stroke-width="0" id="E317-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E317-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E317-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E317-MJMATHI-4C" x="0" y="0"></use><use xlink:href="#E317-MJMAIN-28" x="681" y="0"></use><use xlink:href="#E317-MJMAINB-77" x="1070" y="0"></use><use xlink:href="#E317-MJMAIN-29" x="1901" y="0"></use></g></svg></span><script type="math/tex">L(\bold w)</script><span> 求半梯度，所以将 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="4.13ex" height="2.71ex" viewBox="0 -832.7 1778 1166.9" role="img" focusable="false" style="vertical-align: -0.776ex;"><defs><path stroke-width="0" id="E318-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E318-MJMAIN-31" d="M213 578L200 573Q186 568 160 563T102 556H83V602H102Q149 604 189 617T245 641T273 663Q275 666 285 666Q294 666 302 660V361L303 61Q310 54 315 52T339 48T401 46H427V0H416Q395 3 257 3Q121 3 100 0H88V46H114Q136 46 152 46T177 47T193 50T201 52T207 57T213 61V578Z"></path><path stroke-width="0" id="E318-MJMAIN-30" d="M96 585Q152 666 249 666Q297 666 345 640T423 548Q460 465 460 320Q460 165 417 83Q397 41 362 16T301 -15T250 -22Q224 -22 198 -16T137 16T82 83Q39 165 39 320Q39 494 96 585ZM321 597Q291 629 250 629Q208 629 178 597Q153 571 145 525T137 333Q137 175 145 125T181 46Q209 16 250 16Q290 16 318 46Q347 76 354 130T362 333Q362 478 354 524T321 597Z"></path><path stroke-width="0" id="E318-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><a class="mjx-svg-href" xlink:href="#mjx-eqn-eq%3A10_1"><rect width="1778" height="1000" y="-250" fill="none" stroke="none" pointer-events="all"></rect><g class="MathJax_ref"><use xlink:href="#E318-MJMAIN-28"></use><use xlink:href="#E318-MJMAIN-31" x="389" y="0"></use><use xlink:href="#E318-MJMAIN-30" x="889" y="0"></use><use xlink:href="#E318-MJMAIN-29" x="1389" y="0"></use></g></a></g></svg></span><script type="math/tex">\eqref{eq:10}</script><span> 中的 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="4.666ex" height="2.517ex" viewBox="0 -791.1 2008.9 1083.8" role="img" focusable="false" style="vertical-align: -0.68ex;"><defs><path stroke-width="0" id="E319-MJMATHI-41" d="M208 74Q208 50 254 46Q272 46 272 35Q272 34 270 22Q267 8 264 4T251 0Q249 0 239 0T205 1T141 2Q70 2 50 0H42Q35 7 35 11Q37 38 48 46H62Q132 49 164 96Q170 102 345 401T523 704Q530 716 547 716H555H572Q578 707 578 706L606 383Q634 60 636 57Q641 46 701 46Q726 46 726 36Q726 34 723 22Q720 7 718 4T704 0Q701 0 690 0T651 1T578 2Q484 2 455 0H443Q437 6 437 9T439 27Q443 40 445 43L449 46H469Q523 49 533 63L521 213H283L249 155Q208 86 208 74ZM516 260Q516 271 504 416T490 562L463 519Q447 492 400 412L310 260L413 259Q516 259 516 260Z"></path><path stroke-width="0" id="E319-MJMATHI-74" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path><path stroke-width="0" id="E319-MJMAIN-2B" d="M56 237T56 250T70 270H369V420L370 570Q380 583 389 583Q402 583 409 568V270H707Q722 262 722 250T707 230H409V-68Q401 -82 391 -82H389H387Q375 -82 369 -68V230H70Q56 237 56 250Z"></path><path stroke-width="0" id="E319-MJMAIN-31" d="M213 578L200 573Q186 568 160 563T102 556H83V602H102Q149 604 189 617T245 641T273 663Q275 666 285 666Q294 666 302 660V361L303 61Q310 54 315 52T339 48T401 46H427V0H416Q395 3 257 3Q121 3 100 0H88V46H114Q136 46 152 46T177 47T193 50T201 52T207 57T213 61V578Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E319-MJMATHI-41" x="0" y="0"></use><g transform="translate(750,-150)"><use transform="scale(0.707)" xlink:href="#E319-MJMATHI-74" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E319-MJMAIN-2B" x="361" y="0"></use><use transform="scale(0.707)" xlink:href="#E319-MJMAIN-31" x="1139" y="0"></use></g></g></svg></span><script type="math/tex">A_{t+1}</script><span> 改为 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="28.798ex" height="4.157ex" viewBox="0 -832.7 12399.2 1789.9" role="img" focusable="false" style="vertical-align: -2.223ex;"><defs><path stroke-width="0" id="E320-MJMATHI-41" d="M208 74Q208 50 254 46Q272 46 272 35Q272 34 270 22Q267 8 264 4T251 0Q249 0 239 0T205 1T141 2Q70 2 50 0H42Q35 7 35 11Q37 38 48 46H62Q132 49 164 96Q170 102 345 401T523 704Q530 716 547 716H555H572Q578 707 578 706L606 383Q634 60 636 57Q641 46 701 46Q726 46 726 36Q726 34 723 22Q720 7 718 4T704 0Q701 0 690 0T651 1T578 2Q484 2 455 0H443Q437 6 437 9T439 27Q443 40 445 43L449 46H469Q523 49 533 63L521 213H283L249 155Q208 86 208 74ZM516 260Q516 271 504 416T490 562L463 519Q447 492 400 412L310 260L413 259Q516 259 516 260Z"></path><path stroke-width="0" id="E320-MJMAIN-2217" d="M229 286Q216 420 216 436Q216 454 240 464Q241 464 245 464T251 465Q263 464 273 456T283 436Q283 419 277 356T270 286L328 328Q384 369 389 372T399 375Q412 375 423 365T435 338Q435 325 425 315Q420 312 357 282T289 250L355 219L425 184Q434 175 434 161Q434 146 425 136T401 125Q393 125 383 131T328 171L270 213Q283 79 283 63Q283 53 276 44T250 35Q231 35 224 44T216 63Q216 80 222 143T229 213L171 171Q115 130 110 127Q106 124 100 124Q87 124 76 134T64 161Q64 166 64 169T67 175T72 181T81 188T94 195T113 204T138 215T170 230T210 250L74 315Q65 324 65 338Q65 353 74 363T98 374Q106 374 116 368T171 328L229 286Z"></path><path stroke-width="0" id="E320-MJMATHI-74" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path><path stroke-width="0" id="E320-MJMAIN-2B" d="M56 237T56 250T70 270H369V420L370 570Q380 583 389 583Q402 583 409 568V270H707Q722 262 722 250T707 230H409V-68Q401 -82 391 -82H389H387Q375 -82 369 -68V230H70Q56 237 56 250Z"></path><path stroke-width="0" id="E320-MJMAIN-31" d="M213 578L200 573Q186 568 160 563T102 556H83V602H102Q149 604 189 617T245 641T273 663Q275 666 285 666Q294 666 302 660V361L303 61Q310 54 315 52T339 48T401 46H427V0H416Q395 3 257 3Q121 3 100 0H88V46H114Q136 46 152 46T177 47T193 50T201 52T207 57T213 61V578Z"></path><path stroke-width="0" id="E320-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E320-MJMAIN-61" d="M137 305T115 305T78 320T63 359Q63 394 97 421T218 448Q291 448 336 416T396 340Q401 326 401 309T402 194V124Q402 76 407 58T428 40Q443 40 448 56T453 109V145H493V106Q492 66 490 59Q481 29 455 12T400 -6T353 12T329 54V58L327 55Q325 52 322 49T314 40T302 29T287 17T269 6T247 -2T221 -8T190 -11Q130 -11 82 20T34 107Q34 128 41 147T68 188T116 225T194 253T304 268H318V290Q318 324 312 340Q290 411 215 411Q197 411 181 410T156 406T148 403Q170 388 170 359Q170 334 154 320ZM126 106Q126 75 150 51T209 26Q247 26 276 49T315 109Q317 116 318 175Q318 233 317 233Q309 233 296 232T251 223T193 203T147 166T126 106Z"></path><path stroke-width="0" id="E320-MJMAIN-72" d="M36 46H50Q89 46 97 60V68Q97 77 97 91T98 122T98 161T98 203Q98 234 98 269T98 328L97 351Q94 370 83 376T38 385H20V408Q20 431 22 431L32 432Q42 433 60 434T96 436Q112 437 131 438T160 441T171 442H174V373Q213 441 271 441H277Q322 441 343 419T364 373Q364 352 351 337T313 322Q288 322 276 338T263 372Q263 381 265 388T270 400T273 405Q271 407 250 401Q234 393 226 386Q179 341 179 207V154Q179 141 179 127T179 101T180 81T180 66V61Q181 59 183 57T188 54T193 51T200 49T207 48T216 47T225 47T235 46T245 46H276V0H267Q249 3 140 3Q37 3 28 0H20V46H36Z"></path><path stroke-width="0" id="E320-MJMAIN-67" d="M329 409Q373 453 429 453Q459 453 472 434T485 396Q485 382 476 371T449 360Q416 360 412 390Q410 404 415 411Q415 412 416 414V415Q388 412 363 393Q355 388 355 386Q355 385 359 381T368 369T379 351T388 325T392 292Q392 230 343 187T222 143Q172 143 123 171Q112 153 112 133Q112 98 138 81Q147 75 155 75T227 73Q311 72 335 67Q396 58 431 26Q470 -13 470 -72Q470 -139 392 -175Q332 -206 250 -206Q167 -206 107 -175Q29 -140 29 -75Q29 -39 50 -15T92 18L103 24Q67 55 67 108Q67 155 96 193Q52 237 52 292Q52 355 102 398T223 442Q274 442 318 416L329 409ZM299 343Q294 371 273 387T221 404Q192 404 171 388T145 343Q142 326 142 292Q142 248 149 227T179 192Q196 182 222 182Q244 182 260 189T283 207T294 227T299 242Q302 258 302 292T299 343ZM403 -75Q403 -50 389 -34T348 -11T299 -2T245 0H218Q151 0 138 -6Q118 -15 107 -34T95 -74Q95 -84 101 -97T122 -127T170 -155T250 -167Q319 -167 361 -139T403 -75Z"></path><path stroke-width="0" id="E320-MJMAIN-6D" d="M41 46H55Q94 46 102 60V68Q102 77 102 91T102 122T103 161T103 203Q103 234 103 269T102 328V351Q99 370 88 376T43 385H25V408Q25 431 27 431L37 432Q47 433 65 434T102 436Q119 437 138 438T167 441T178 442H181V402Q181 364 182 364T187 369T199 384T218 402T247 421T285 437Q305 442 336 442Q351 442 364 440T387 434T406 426T421 417T432 406T441 395T448 384T452 374T455 366L457 361L460 365Q463 369 466 373T475 384T488 397T503 410T523 422T546 432T572 439T603 442Q729 442 740 329Q741 322 741 190V104Q741 66 743 59T754 49Q775 46 803 46H819V0H811L788 1Q764 2 737 2T699 3Q596 3 587 0H579V46H595Q656 46 656 62Q657 64 657 200Q656 335 655 343Q649 371 635 385T611 402T585 404Q540 404 506 370Q479 343 472 315T464 232V168V108Q464 78 465 68T468 55T477 49Q498 46 526 46H542V0H534L510 1Q487 2 460 2T422 3Q319 3 310 0H302V46H318Q379 46 379 62Q380 64 380 200Q379 335 378 343Q372 371 358 385T334 402T308 404Q263 404 229 370Q202 343 195 315T187 232V168V108Q187 78 188 68T191 55T200 49Q221 46 249 46H265V0H257L234 1Q210 2 183 2T145 3Q42 3 33 0H25V46H41Z"></path><path stroke-width="0" id="E320-MJMAIN-78" d="M201 0Q189 3 102 3Q26 3 17 0H11V46H25Q48 47 67 52T96 61T121 78T139 96T160 122T180 150L226 210L168 288Q159 301 149 315T133 336T122 351T113 363T107 370T100 376T94 379T88 381T80 383Q74 383 44 385H16V431H23Q59 429 126 429Q219 429 229 431H237V385Q201 381 201 369Q201 367 211 353T239 315T268 274L272 270L297 304Q329 345 329 358Q329 364 327 369T322 376T317 380T310 384L307 385H302V431H309Q324 428 408 428Q487 428 493 431H499V385H492Q443 385 411 368Q394 360 377 341T312 257L296 236L358 151Q424 61 429 57T446 50Q464 46 499 46H516V0H510H502Q494 1 482 1T457 2T432 2T414 3Q403 3 377 3T327 1L304 0H295V46H298Q309 46 320 51T331 63Q331 65 291 120L250 175Q249 174 219 133T185 88Q181 83 181 74Q181 63 188 55T206 46Q208 46 208 23V0H201Z"></path><path stroke-width="0" id="E320-MJMATHI-61" d="M33 157Q33 258 109 349T280 441Q331 441 370 392Q386 422 416 422Q429 422 439 414T449 394Q449 381 412 234T374 68Q374 43 381 35T402 26Q411 27 422 35Q443 55 463 131Q469 151 473 152Q475 153 483 153H487Q506 153 506 144Q506 138 501 117T481 63T449 13Q436 0 417 -8Q409 -10 393 -10Q359 -10 336 5T306 36L300 51Q299 52 296 50Q294 48 292 46Q233 -10 172 -10Q117 -10 75 30T33 157ZM351 328Q351 334 346 350T323 385T277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q217 26 254 59T298 110Q300 114 325 217T351 328Z"></path><path stroke-width="0" id="E320-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E320-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E320-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E320-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E320-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E320-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E320-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E320-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E320-MJMAIN-2217" x="1060" y="452"></use><g transform="translate(750,-307)"><use transform="scale(0.707)" xlink:href="#E320-MJMATHI-74" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E320-MJMAIN-2B" x="361" y="0"></use><use transform="scale(0.707)" xlink:href="#E320-MJMAIN-31" x="1139" y="0"></use></g><use xlink:href="#E320-MJMAIN-3D" x="2286" y="0"></use><g transform="translate(3342,0)"><use xlink:href="#E320-MJMAIN-61"></use><use xlink:href="#E320-MJMAIN-72" x="500" y="0"></use><use xlink:href="#E320-MJMAIN-67" x="892" y="0"></use><g transform="translate(1558,0)"><use xlink:href="#E320-MJMAIN-6D"></use><use xlink:href="#E320-MJMAIN-61" x="833" y="0"></use><use xlink:href="#E320-MJMAIN-78" x="1333" y="0"></use></g><use transform="scale(0.707)" xlink:href="#E320-MJMATHI-61" x="2153" y="-1140"></use></g><use xlink:href="#E320-MJMATHI-71" x="7039" y="0"></use><use xlink:href="#E320-MJMAIN-28" x="7499" y="0"></use><g transform="translate(7888,0)"><use xlink:href="#E320-MJMATHI-53" x="0" y="0"></use><g transform="translate(613,-150)"><use transform="scale(0.707)" xlink:href="#E320-MJMATHI-74" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E320-MJMAIN-2B" x="361" y="0"></use><use transform="scale(0.707)" xlink:href="#E320-MJMAIN-31" x="1139" y="0"></use></g></g><use xlink:href="#E320-MJMAIN-2C" x="9760" y="0"></use><use xlink:href="#E320-MJMATHI-61" x="10205" y="0"></use><use xlink:href="#E320-MJMAIN-3B" x="10734" y="0"></use><use xlink:href="#E320-MJMAINB-77" x="11179" y="0"></use><use xlink:href="#E320-MJMAIN-29" x="12010" y="0"></use></g></svg></span><script type="math/tex">\displaystyle A_{t+1}^* = \underset{a}{\arg\max}\; q(S_{t+1},a;\bold w)</script><span> 即可得到解为：</span></p><div contenteditable="false" spellcheck="false" class="mathjax-block md-end-block md-math-block md-rawblock" id="mathjax-n45" cid="n45" mdtype="math_block"><div class="md-rawblock-container md-math-container" tabindex="-1"><div class="MathJax_SVG_Display"><span class="MathJax_SVG" id="MathJax-Element-255-Frame" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="98.296ex" height="7.726ex" viewBox="0 -1912.5 42321.7 3326.6" role="img" focusable="false" style="vertical-align: -3.284ex; max-width: 100%;"><defs><path stroke-width="0" id="E282-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E282-MJMAIN-31" d="M213 578L200 573Q186 568 160 563T102 556H83V602H102Q149 604 189 617T245 641T273 663Q275 666 285 666Q294 666 302 660V361L303 61Q310 54 315 52T339 48T401 46H427V0H416Q395 3 257 3Q121 3 100 0H88V46H114Q136 46 152 46T177 47T193 50T201 52T207 57T213 61V578Z"></path><path stroke-width="0" id="E282-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E282-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E282-MJMATHI-4C" d="M228 637Q194 637 192 641Q191 643 191 649Q191 673 202 682Q204 683 217 683Q271 680 344 680Q485 680 506 683H518Q524 677 524 674T522 656Q517 641 513 637H475Q406 636 394 628Q387 624 380 600T313 336Q297 271 279 198T252 88L243 52Q243 48 252 48T311 46H328Q360 46 379 47T428 54T478 72T522 106T564 161Q580 191 594 228T611 270Q616 273 628 273H641Q647 264 647 262T627 203T583 83T557 9Q555 4 553 3T537 0T494 -1Q483 -1 418 -1T294 0H116Q32 0 32 10Q32 17 34 24Q39 43 44 45Q48 46 59 46H65Q92 46 125 49Q139 52 144 61Q147 65 216 339T285 628Q285 635 228 637Z"></path><path stroke-width="0" id="E282-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E282-MJMATHI-54" d="M40 437Q21 437 21 445Q21 450 37 501T71 602L88 651Q93 669 101 677H569H659Q691 677 697 676T704 667Q704 661 687 553T668 444Q668 437 649 437Q640 437 637 437T631 442L629 445Q629 451 635 490T641 551Q641 586 628 604T573 629Q568 630 515 631Q469 631 457 630T439 622Q438 621 368 343T298 60Q298 48 386 46Q418 46 427 45T436 36Q436 31 433 22Q429 4 424 1L422 0Q419 0 415 0Q410 0 363 1T228 2Q99 2 64 0H49Q43 6 43 9T45 27Q49 40 55 46H83H94Q174 46 189 55Q190 56 191 56Q196 59 201 76T241 233Q258 301 269 344Q339 619 339 625Q339 630 310 630H279Q212 630 191 624Q146 614 121 583T67 467Q60 445 57 441T43 437H40Z"></path><path stroke-width="0" id="E282-MJMATHI-44" d="M287 628Q287 635 230 637Q207 637 200 638T193 647Q193 655 197 667T204 682Q206 683 403 683Q570 682 590 682T630 676Q702 659 752 597T803 431Q803 275 696 151T444 3L430 1L236 0H125H72Q48 0 41 2T33 11Q33 13 36 25Q40 41 44 43T67 46Q94 46 127 49Q141 52 146 61Q149 65 218 339T287 628ZM703 469Q703 507 692 537T666 584T629 613T590 629T555 636Q553 636 541 636T512 636T479 637H436Q392 637 386 627Q384 623 313 339T242 52Q242 48 253 48T330 47Q335 47 349 47T373 46Q499 46 581 128Q617 164 640 212T683 339T703 469Z"></path><path stroke-width="0" id="E282-MJMATHI-51" d="M399 -80Q399 -47 400 -30T402 -11V-7L387 -11Q341 -22 303 -22Q208 -22 138 35T51 201Q50 209 50 244Q50 346 98 438T227 601Q351 704 476 704Q514 704 524 703Q621 689 680 617T740 435Q740 255 592 107Q529 47 461 16L444 8V3Q444 2 449 -24T470 -66T516 -82Q551 -82 583 -60T625 -3Q631 11 638 11Q647 11 649 2Q649 -6 639 -34T611 -100T557 -165T481 -194Q399 -194 399 -87V-80ZM636 468Q636 523 621 564T580 625T530 655T477 665Q429 665 379 640Q277 591 215 464T153 216Q153 110 207 59Q231 38 236 38V46Q236 86 269 120T347 155Q372 155 390 144T417 114T429 82T435 55L448 64Q512 108 557 185T619 334T636 468ZM314 18Q362 18 404 39L403 49Q399 104 366 115Q354 117 347 117Q344 117 341 117T337 118Q317 118 296 98T274 52Q274 18 314 18Z"></path><path stroke-width="0" id="E282-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E282-MJSZ2-2211" d="M60 948Q63 950 665 950H1267L1325 815Q1384 677 1388 669H1348L1341 683Q1320 724 1285 761Q1235 809 1174 838T1033 881T882 898T699 902H574H543H251L259 891Q722 258 724 252Q725 250 724 246Q721 243 460 -56L196 -356Q196 -357 407 -357Q459 -357 548 -357T676 -358Q812 -358 896 -353T1063 -332T1204 -283T1307 -196Q1328 -170 1348 -124H1388Q1388 -125 1381 -145T1356 -210T1325 -294L1267 -449L666 -450Q64 -450 61 -448Q55 -446 55 -439Q55 -437 57 -433L590 177Q590 178 557 222T452 366T322 544L56 909L55 924Q55 945 60 948Z"></path><path stroke-width="0" id="E282-MJMATHI-74" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path><path stroke-width="0" id="E282-MJMAINB-78" d="M227 0Q212 3 121 3Q40 3 28 0H21V62H117L245 213L109 382H26V444H34Q49 441 143 441Q247 441 265 444H274V382H246L281 339Q315 297 316 297Q320 297 354 341L389 382H352V444H360Q375 441 466 441Q547 441 559 444H566V382H471L355 246L504 63L545 62H586V0H578Q563 3 469 3Q365 3 347 0H338V62H366Q366 63 326 112T285 163L198 63L217 62H235V0H227Z"></path><path stroke-width="0" id="E282-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E282-MJMATHI-41" d="M208 74Q208 50 254 46Q272 46 272 35Q272 34 270 22Q267 8 264 4T251 0Q249 0 239 0T205 1T141 2Q70 2 50 0H42Q35 7 35 11Q37 38 48 46H62Q132 49 164 96Q170 102 345 401T523 704Q530 716 547 716H555H572Q578 707 578 706L606 383Q634 60 636 57Q641 46 701 46Q726 46 726 36Q726 34 723 22Q720 7 718 4T704 0Q701 0 690 0T651 1T578 2Q484 2 455 0H443Q437 6 437 9T439 27Q443 40 445 43L449 46H469Q523 49 533 63L521 213H283L249 155Q208 86 208 74ZM516 260Q516 271 504 416T490 562L463 519Q447 492 400 412L310 260L413 259Q516 259 516 260Z"></path><path stroke-width="0" id="E282-MJMATHI-78" d="M52 289Q59 331 106 386T222 442Q257 442 286 424T329 379Q371 442 430 442Q467 442 494 420T522 361Q522 332 508 314T481 292T458 288Q439 288 427 299T415 328Q415 374 465 391Q454 404 425 404Q412 404 406 402Q368 386 350 336Q290 115 290 78Q290 50 306 38T341 26Q378 26 414 59T463 140Q466 150 469 151T485 153H489Q504 153 504 145Q504 144 502 134Q486 77 440 33T333 -11Q263 -11 227 52Q186 -10 133 -10H127Q78 -10 57 16T35 71Q35 103 54 123T99 143Q142 143 142 101Q142 81 130 66T107 46T94 41L91 40Q91 39 97 36T113 29T132 26Q168 26 194 71Q203 87 217 139T245 247T261 313Q266 340 266 352Q266 380 251 392T217 404Q177 404 142 372T93 290Q91 281 88 280T72 278H58Q52 284 52 289Z"></path><path stroke-width="0" id="E282-MJMAIN-2212" d="M84 237T84 250T98 270H679Q694 262 694 250T679 230H98Q84 237 84 250Z"></path><path stroke-width="0" id="E282-MJMATHI-3B3" d="M31 249Q11 249 11 258Q11 275 26 304T66 365T129 418T206 441Q233 441 239 440Q287 429 318 386T371 255Q385 195 385 170Q385 166 386 166L398 193Q418 244 443 300T486 391T508 430Q510 431 524 431H537Q543 425 543 422Q543 418 522 378T463 251T391 71Q385 55 378 6T357 -100Q341 -165 330 -190T303 -216Q286 -216 286 -188Q286 -138 340 32L346 51L347 69Q348 79 348 100Q348 257 291 317Q251 355 196 355Q148 355 108 329T51 260Q49 251 47 251Q45 249 31 249Z"></path><path stroke-width="0" id="E282-MJMAIN-2B" d="M56 237T56 250T70 270H369V420L370 570Q380 583 389 583Q402 583 409 568V270H707Q722 262 722 250T707 230H409V-68Q401 -82 391 -82H389H387Q375 -82 369 -68V230H70Q56 237 56 250Z"></path><path stroke-width="0" id="E282-MJMAIN-2217" d="M229 286Q216 420 216 436Q216 454 240 464Q241 464 245 464T251 465Q263 464 273 456T283 436Q283 419 277 356T270 286L328 328Q384 369 389 372T399 375Q412 375 423 365T435 338Q435 325 425 315Q420 312 357 282T289 250L355 219L425 184Q434 175 434 161Q434 146 425 136T401 125Q393 125 383 131T328 171L270 213Q283 79 283 63Q283 53 276 44T250 35Q231 35 224 44T216 63Q216 80 222 143T229 213L171 171Q115 130 110 127Q106 124 100 124Q87 124 76 134T64 161Q64 166 64 169T67 175T72 181T81 188T94 195T113 204T138 215T170 230T210 250L74 315Q65 324 65 338Q65 353 74 363T98 374Q106 374 116 368T171 328L229 286Z"></path><path stroke-width="0" id="E282-MJSZ4-28" d="M758 -1237T758 -1240T752 -1249H736Q718 -1249 717 -1248Q711 -1245 672 -1199Q237 -706 237 251T672 1700Q697 1730 716 1749Q718 1750 735 1750H752Q758 1744 758 1741Q758 1737 740 1713T689 1644T619 1537T540 1380T463 1176Q348 802 348 251Q348 -242 441 -599T744 -1218Q758 -1237 758 -1240Z"></path><path stroke-width="0" id="E282-MJSZ4-29" d="M33 1741Q33 1750 51 1750H60H65Q73 1750 81 1743T119 1700Q554 1207 554 251Q554 -707 119 -1199Q76 -1250 66 -1250Q65 -1250 62 -1250T56 -1249Q55 -1249 53 -1249T49 -1250Q33 -1250 33 -1239Q33 -1236 50 -1214T98 -1150T163 -1052T238 -910T311 -727Q443 -335 443 251Q443 402 436 532T405 831T339 1142T224 1438T50 1716Q33 1737 33 1741Z"></path><path stroke-width="0" id="E282-MJMATHI-52" d="M230 637Q203 637 198 638T193 649Q193 676 204 682Q206 683 378 683Q550 682 564 680Q620 672 658 652T712 606T733 563T739 529Q739 484 710 445T643 385T576 351T538 338L545 333Q612 295 612 223Q612 212 607 162T602 80V71Q602 53 603 43T614 25T640 16Q668 16 686 38T712 85Q717 99 720 102T735 105Q755 105 755 93Q755 75 731 36Q693 -21 641 -21H632Q571 -21 531 4T487 82Q487 109 502 166T517 239Q517 290 474 313Q459 320 449 321T378 323H309L277 193Q244 61 244 59Q244 55 245 54T252 50T269 48T302 46H333Q339 38 339 37T336 19Q332 6 326 0H311Q275 2 180 2Q146 2 117 2T71 2T50 1Q33 1 33 10Q33 12 36 24Q41 43 46 45Q50 46 61 46H67Q94 46 127 49Q141 52 146 61Q149 65 218 339T287 628Q287 635 230 637ZM630 554Q630 586 609 608T523 636Q521 636 500 636T462 637H440Q393 637 386 627Q385 624 352 494T319 361Q319 360 388 360Q466 361 492 367Q556 377 592 426Q608 449 619 486T630 554Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><g transform="translate(40543,0)"><g id="mjx-eqn-11" transform="translate(0,-99)"><use xlink:href="#E282-MJMAIN-28"></use><use xlink:href="#E282-MJMAIN-31" x="389" y="0"></use><use xlink:href="#E282-MJMAIN-31" x="889" y="0"></use><use xlink:href="#E282-MJMAIN-29" x="1389" y="0"></use></g></g><g transform="translate(4654,0)"><g transform="translate(-19,0)"><g transform="translate(0,-99)"><use xlink:href="#E282-MJMAINB-77" x="0" y="0"></use><g transform="translate(831,-155)"><use transform="scale(0.707)" xlink:href="#E282-MJMATHI-4C" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E282-MJMATHI-53" x="681" y="0"></use><use transform="scale(0.707)" xlink:href="#E282-MJMATHI-54" x="1326" y="0"></use><use transform="scale(0.707)" xlink:href="#E282-MJMATHI-44" x="2029" y="0"></use><use transform="scale(0.707)" xlink:href="#E282-MJMATHI-51" x="2858" y="0"></use></g><use xlink:href="#E282-MJMAIN-3D" x="3789" y="0"></use><g transform="translate(4844,0)"><use xlink:href="#E282-MJSZ4-28"></use><g transform="translate(792,0)"><use xlink:href="#E282-MJSZ2-2211" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E282-MJMATHI-74" x="840" y="-1499"></use></g><use xlink:href="#E282-MJMAINB-78" x="2402" y="0"></use><use xlink:href="#E282-MJMAIN-28" x="3009" y="0"></use><g transform="translate(3398,0)"><use xlink:href="#E282-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E282-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E282-MJMAIN-2C" x="4366" y="0"></use><g transform="translate(4811,0)"><use xlink:href="#E282-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E282-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E282-MJMAIN-29" x="5916" y="0"></use><use xlink:href="#E282-MJMAIN-28" x="6305" y="0"></use><use xlink:href="#E282-MJMATHI-78" x="6694" y="0"></use><use xlink:href="#E282-MJMAIN-28" x="7266" y="0"></use><g transform="translate(7655,0)"><use xlink:href="#E282-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E282-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E282-MJMAIN-2C" x="8624" y="0"></use><g transform="translate(9068,0)"><use xlink:href="#E282-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E282-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E282-MJMAIN-29" x="10174" y="0"></use><use xlink:href="#E282-MJMAIN-2212" x="10785" y="0"></use><use xlink:href="#E282-MJMATHI-3B3" x="11785" y="0"></use><use xlink:href="#E282-MJMAINB-78" x="12328" y="0"></use><use xlink:href="#E282-MJMAIN-28" x="12935" y="0"></use><g transform="translate(13324,0)"><use xlink:href="#E282-MJMATHI-53" x="0" y="0"></use><g transform="translate(613,-150)"><use transform="scale(0.707)" xlink:href="#E282-MJMATHI-74" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E282-MJMAIN-2B" x="361" y="0"></use><use transform="scale(0.707)" xlink:href="#E282-MJMAIN-31" x="1139" y="0"></use></g></g><use xlink:href="#E282-MJMAIN-2C" x="15196" y="0"></use><g transform="translate(15641,0)"><use xlink:href="#E282-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E282-MJMAIN-2217" x="1060" y="452"></use><g transform="translate(750,-307)"><use transform="scale(0.707)" xlink:href="#E282-MJMATHI-74" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E282-MJMAIN-2B" x="361" y="0"></use><use transform="scale(0.707)" xlink:href="#E282-MJMAIN-31" x="1139" y="0"></use></g></g><use xlink:href="#E282-MJMAIN-29" x="17650" y="0"></use><g transform="translate(18039,0)"><use xlink:href="#E282-MJMAIN-29" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E282-MJMATHI-54" x="550" y="583"></use></g><use xlink:href="#E282-MJSZ4-29" x="19025" y="0"></use><g transform="translate(19817,1476)"><use transform="scale(0.707)" xlink:href="#E282-MJMAIN-2212" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E282-MJMAIN-31" x="778" y="0"></use></g></g><g transform="translate(25833,0)"><use xlink:href="#E282-MJSZ2-2211" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E282-MJMATHI-74" x="840" y="-1499"></use></g><g transform="translate(27443,0)"><use xlink:href="#E282-MJMATHI-52" x="0" y="0"></use><g transform="translate(759,-150)"><use transform="scale(0.707)" xlink:href="#E282-MJMATHI-74" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E282-MJMAIN-2B" x="361" y="0"></use><use transform="scale(0.707)" xlink:href="#E282-MJMAIN-31" x="1139" y="0"></use></g></g><use xlink:href="#E282-MJMAINB-78" x="29461" y="0"></use><use xlink:href="#E282-MJMAIN-28" x="30068" y="0"></use><g transform="translate(30457,0)"><use xlink:href="#E282-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E282-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E282-MJMAIN-2C" x="31425" y="0"></use><g transform="translate(31870,0)"><use xlink:href="#E282-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E282-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E282-MJMAIN-29" x="32975" y="0"></use></g></g></g></g></svg></span></div><script type="math/tex; mode=display" id="MathJax-Element-255">\bold w_{LSTDQ} = \left(\sum_t \bold x(S_t,A_t)(x(S_t,A_t) - \gamma\bold x(S_{t+1},A_{t+1}^*))^T \right)^{-1} \sum_t R_{t+1}\bold x(S_t,A_t)</script></div></div><p><span>那么有基于 Q 学习的最小二乘最优策略求解算法：</span></p><div contenteditable="false" spellcheck="false" class="mathjax-block md-end-block md-math-block md-rawblock" id="mathjax-n47" cid="n47" mdtype="math_block"><div class="md-rawblock-container md-math-container" tabindex="-1"><div class="MathJax_SVG_Display"><span class="MathJax_SVG" id="MathJax-Element-256-Frame" tabindex="-1" style="font-size: 100%; display: inline-block; zoom: 0.971345;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="101.294ex" height="50.459ex" viewBox="-18.1 -43.5 43612.5 21725.5" role="img" focusable="false" style="vertical-align: -50.358ex; margin-left: -0.042ex; max-width: 100%;"><defs><path stroke-width="0" id="E283-MJMAINB-36" d="M48 318Q48 395 68 456T120 553T193 613T273 646T350 655Q425 655 461 616T497 524Q497 485 475 468T428 451Q399 451 378 470T357 521Q357 565 403 588Q375 601 351 601Q313 601 282 584Q242 565 222 526Q199 473 199 367Q201 369 210 380T227 396T246 410T275 422T312 426Q438 426 494 332Q526 285 526 208V199Q526 112 465 53Q428 17 388 3T285 -11Q236 -11 195 7T135 43T104 80Q48 165 48 318ZM375 231V244V268Q375 295 373 310T364 342T341 366T299 374H297Q231 374 208 287Q200 257 200 196Q201 120 209 100Q231 47 288 47Q351 47 368 90Q375 112 375 231Z"></path><path stroke-width="0" id="E283-MJMAINB-2D" d="M13 166V278H318V166H13Z"></path><path stroke-width="0" id="E283-MJMAINB-37" d="M256 -11Q231 -11 208 5T185 65Q185 105 193 146T212 220T241 289T275 349T312 402T346 445T377 479T397 502L400 504H301Q156 503 150 497Q142 491 134 456T126 407H64V411Q65 414 82 544T99 675T130 676H161V673Q161 669 162 666T167 661T173 657T181 654T190 652T200 651T210 650T220 649T229 648Q237 648 254 647T276 646Q277 646 426 644H558V620V607Q558 596 551 586T509 537Q489 515 476 500Q390 401 384 393Q349 339 337 259T324 113T322 38Q307 -11 256 -11Z"></path><path stroke-width="0" id="E283-MJMAINB-51" d="M64 339Q64 431 96 502T182 614T295 675T420 696Q469 696 481 695Q620 680 709 589T798 339Q798 255 768 184Q720 77 611 26L600 21Q635 -26 682 -26H696Q769 -26 769 0Q769 7 774 12T787 18Q805 18 805 -7V-13Q803 -64 785 -106T737 -171Q720 -183 697 -191Q687 -193 668 -193Q636 -193 613 -182T575 -144T552 -94T532 -27Q531 -23 530 -16T528 -6T526 -3L512 -5Q499 -7 477 -8T431 -10Q393 -10 382 -9Q238 8 151 97T64 339ZM326 80Q326 113 356 138T430 163Q492 163 542 100L553 86Q554 85 561 91T578 108Q637 179 637 330Q637 430 619 498T548 604Q500 641 425 641Q408 641 390 637T347 623T299 590T259 535Q226 469 226 338Q226 244 246 180T318 79L325 74Q326 74 326 80ZM506 58Q480 112 433 112Q412 112 395 104T378 77Q378 44 431 44Q480 44 506 58Z"></path><path stroke-width="0" id="E283-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E283-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E283-MJMATHI-73" d="M131 289Q131 321 147 354T203 415T300 442Q362 442 390 415T419 355Q419 323 402 308T364 292Q351 292 340 300T328 326Q328 342 337 354T354 372T367 378Q368 378 368 379Q368 382 361 388T336 399T297 405Q249 405 227 379T204 326Q204 301 223 291T278 274T330 259Q396 230 396 163Q396 135 385 107T352 51T289 7T195 -10Q118 -10 86 19T53 87Q53 126 74 143T118 160Q133 160 146 151T160 120Q160 94 142 76T111 58Q109 57 108 57T107 55Q108 52 115 47T146 34T201 27Q237 27 263 38T301 66T318 97T323 122Q323 150 302 164T254 181T195 196T148 231Q131 256 131 289Z"></path><path stroke-width="0" id="E283-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E283-MJMATHI-61" d="M33 157Q33 258 109 349T280 441Q331 441 370 392Q386 422 416 422Q429 422 439 414T449 394Q449 381 412 234T374 68Q374 43 381 35T402 26Q411 27 422 35Q443 55 463 131Q469 151 473 152Q475 153 483 153H487Q506 153 506 144Q506 138 501 117T481 63T449 13Q436 0 417 -8Q409 -10 393 -10Q359 -10 336 5T306 36L300 51Q299 52 296 50Q294 48 292 46Q233 -10 172 -10Q117 -10 75 30T33 157ZM351 328Q351 334 346 350T323 385T277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q217 26 254 59T298 110Q300 114 325 217T351 328Z"></path><path stroke-width="0" id="E283-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E283-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E283-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E283-MJMAIN-2208" d="M84 250Q84 372 166 450T360 539Q361 539 377 539T419 540T469 540H568Q583 532 583 520Q583 511 570 501L466 500Q355 499 329 494Q280 482 242 458T183 409T147 354T129 306T124 272V270H568Q583 262 583 250T568 230H124V228Q124 207 134 177T167 112T231 48T328 7Q355 1 466 0H570Q583 -10 583 -20Q583 -32 568 -40H471Q464 -40 446 -40T417 -41Q262 -41 172 45Q84 127 84 250Z"></path><path stroke-width="0" id="E283-MJCAL-53" d="M554 512Q536 512 536 522Q536 525 539 539T542 564Q542 588 528 604Q515 616 482 625T410 635Q374 635 349 624T312 594T295 561T290 532Q290 505 303 482T342 442T378 419T409 404Q435 391 451 383T494 357T535 323T562 282T574 231Q574 133 464 56T220 -22Q138 -22 78 21T18 123Q18 184 61 227T156 274Q178 274 178 263Q178 260 177 258Q172 247 164 239T151 227T136 218L127 213L124 202Q118 186 118 163Q120 124 165 86T292 48Q374 48 423 86T473 186V193Q473 267 347 327Q268 364 239 389Q191 431 191 486Q191 547 242 600T356 679T470 705Q472 705 478 705T489 704Q551 704 596 682T642 610Q642 566 621 545Q592 516 554 512Z"></path><path stroke-width="0" id="E283-MJCAL-41" d="M576 668Q576 688 606 708T660 728Q676 728 675 712V571Q675 409 688 252Q696 122 720 57Q722 53 723 50T728 46T732 43T737 41T743 39L754 45Q788 61 803 61Q819 61 819 47Q818 43 814 35Q799 15 755 -7T675 -30Q659 -30 648 -25T630 -8T621 11T614 34Q603 77 599 106T594 146T591 160V163H460L329 164L316 145Q241 35 196 -7T119 -50T59 -24T30 43Q30 75 46 100T74 125Q81 125 83 120T88 104T96 84Q118 57 151 57Q189 57 277 182Q432 400 542 625L559 659H567Q574 659 575 660T576 668ZM584 249Q579 333 577 386T575 473T574 520V581L563 560Q497 426 412 290L372 228L370 224H371L383 228L393 232H586L584 249Z"></path><path stroke-width="0" id="E283-MJMATHI-3C0" d="M132 -11Q98 -11 98 22V33L111 61Q186 219 220 334L228 358H196Q158 358 142 355T103 336Q92 329 81 318T62 297T53 285Q51 284 38 284Q19 284 19 294Q19 300 38 329T93 391T164 429Q171 431 389 431Q549 431 553 430Q573 423 573 402Q573 371 541 360Q535 358 472 358H408L405 341Q393 269 393 222Q393 170 402 129T421 65T431 37Q431 20 417 5T381 -10Q370 -10 363 -7T347 17T331 77Q330 86 330 121Q330 170 339 226T357 318T367 358H269L268 354Q268 351 249 275T206 114T175 17Q164 -11 132 -11Z"></path><path stroke-width="0" id="E283-MJMAIN-31" d="M213 578L200 573Q186 568 160 563T102 556H83V602H102Q149 604 189 617T245 641T273 663Q275 666 285 666Q294 666 302 660V361L303 61Q310 54 315 52T339 48T401 46H427V0H416Q395 3 257 3Q121 3 100 0H88V46H114Q136 46 152 46T177 47T193 50T201 52T207 57T213 61V578Z"></path><path stroke-width="0" id="E283-MJMAIN-2E" d="M78 60Q78 84 95 102T138 120Q162 120 180 104T199 61Q199 36 182 18T139 0T96 17T78 60Z"></path><path stroke-width="0" id="E283-MJMAIN-2190" d="M944 261T944 250T929 230H165Q167 228 182 216T211 189T244 152T277 96T303 25Q308 7 308 0Q308 -11 288 -11Q281 -11 278 -11T272 -7T267 2T263 21Q245 94 195 151T73 236Q58 242 55 247Q55 254 59 257T73 264Q121 283 158 314T215 375T247 434T264 480L267 497Q269 503 270 505T275 509T288 511Q308 511 308 500Q308 493 303 475Q293 438 278 406T246 352T215 315T185 287T165 270H929Q944 261 944 250Z"></path><path stroke-width="0" id="E283-MJMAIN-32" d="M109 429Q82 429 66 447T50 491Q50 562 103 614T235 666Q326 666 387 610T449 465Q449 422 429 383T381 315T301 241Q265 210 201 149L142 93L218 92Q375 92 385 97Q392 99 409 186V189H449V186Q448 183 436 95T421 3V0H50V19V31Q50 38 56 46T86 81Q115 113 136 137Q145 147 170 174T204 211T233 244T261 278T284 308T305 340T320 369T333 401T340 431T343 464Q343 527 309 573T212 619Q179 619 154 602T119 569T109 550Q109 549 114 549Q132 549 151 535T170 489Q170 464 154 447T109 429Z"></path><path stroke-width="0" id="E283-MJMAIN-2032" d="M79 43Q73 43 52 49T30 61Q30 68 85 293T146 528Q161 560 198 560Q218 560 240 545T262 501Q262 496 260 486Q259 479 173 263T84 45T79 43Z"></path><path stroke-width="0" id="E283-MJSZ2-2211" d="M60 948Q63 950 665 950H1267L1325 815Q1384 677 1388 669H1348L1341 683Q1320 724 1285 761Q1235 809 1174 838T1033 881T882 898T699 902H574H543H251L259 891Q722 258 724 252Q725 250 724 246Q721 243 460 -56L196 -356Q196 -357 407 -357Q459 -357 548 -357T676 -358Q812 -358 896 -353T1063 -332T1204 -283T1307 -196Q1328 -170 1348 -124H1388Q1388 -125 1381 -145T1356 -210T1325 -294L1267 -449L666 -450Q64 -450 61 -448Q55 -446 55 -439Q55 -437 57 -433L590 177Q590 178 557 222T452 366T322 544L56 909L55 924Q55 945 60 948Z"></path><path stroke-width="0" id="E283-MJMATHI-74" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path><path stroke-width="0" id="E283-MJMAINB-78" d="M227 0Q212 3 121 3Q40 3 28 0H21V62H117L245 213L109 382H26V444H34Q49 441 143 441Q247 441 265 444H274V382H246L281 339Q315 297 316 297Q320 297 354 341L389 382H352V444H360Q375 441 466 441Q547 441 559 444H566V382H471L355 246L504 63L545 62H586V0H578Q563 3 469 3Q365 3 347 0H338V62H366Q366 63 326 112T285 163L198 63L217 62H235V0H227Z"></path><path stroke-width="0" id="E283-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E283-MJMATHI-41" d="M208 74Q208 50 254 46Q272 46 272 35Q272 34 270 22Q267 8 264 4T251 0Q249 0 239 0T205 1T141 2Q70 2 50 0H42Q35 7 35 11Q37 38 48 46H62Q132 49 164 96Q170 102 345 401T523 704Q530 716 547 716H555H572Q578 707 578 706L606 383Q634 60 636 57Q641 46 701 46Q726 46 726 36Q726 34 723 22Q720 7 718 4T704 0Q701 0 690 0T651 1T578 2Q484 2 455 0H443Q437 6 437 9T439 27Q443 40 445 43L449 46H469Q523 49 533 63L521 213H283L249 155Q208 86 208 74ZM516 260Q516 271 504 416T490 562L463 519Q447 492 400 412L310 260L413 259Q516 259 516 260Z"></path><path stroke-width="0" id="E283-MJMAIN-2212" d="M84 237T84 250T98 270H679Q694 262 694 250T679 230H98Q84 237 84 250Z"></path><path stroke-width="0" id="E283-MJMATHI-3B3" d="M31 249Q11 249 11 258Q11 275 26 304T66 365T129 418T206 441Q233 441 239 440Q287 429 318 386T371 255Q385 195 385 170Q385 166 386 166L398 193Q418 244 443 300T486 391T508 430Q510 431 524 431H537Q543 425 543 422Q543 418 522 378T463 251T391 71Q385 55 378 6T357 -100Q341 -165 330 -190T303 -216Q286 -216 286 -188Q286 -138 340 32L346 51L347 69Q348 79 348 100Q348 257 291 317Q251 355 196 355Q148 355 108 329T51 260Q49 251 47 251Q45 249 31 249Z"></path><path stroke-width="0" id="E283-MJMAIN-2B" d="M56 237T56 250T70 270H369V420L370 570Q380 583 389 583Q402 583 409 568V270H707Q722 262 722 250T707 230H409V-68Q401 -82 391 -82H389H387Q375 -82 369 -68V230H70Q56 237 56 250Z"></path><path stroke-width="0" id="E283-MJMAIN-2217" d="M229 286Q216 420 216 436Q216 454 240 464Q241 464 245 464T251 465Q263 464 273 456T283 436Q283 419 277 356T270 286L328 328Q384 369 389 372T399 375Q412 375 423 365T435 338Q435 325 425 315Q420 312 357 282T289 250L355 219L425 184Q434 175 434 161Q434 146 425 136T401 125Q393 125 383 131T328 171L270 213Q283 79 283 63Q283 53 276 44T250 35Q231 35 224 44T216 63Q216 80 222 143T229 213L171 171Q115 130 110 127Q106 124 100 124Q87 124 76 134T64 161Q64 166 64 169T67 175T72 181T81 188T94 195T113 204T138 215T170 230T210 250L74 315Q65 324 65 338Q65 353 74 363T98 374Q106 374 116 368T171 328L229 286Z"></path><path stroke-width="0" id="E283-MJSZ1-28" d="M152 251Q152 646 388 850H416Q422 844 422 841Q422 837 403 816T357 753T302 649T255 482T236 250Q236 124 255 19T301 -147T356 -251T403 -315T422 -340Q422 -343 416 -349H388Q359 -325 332 -296T271 -213T212 -97T170 56T152 251Z"></path><path stroke-width="0" id="E283-MJSZ1-29" d="M305 251Q305 -145 69 -349H56Q43 -349 39 -347T35 -338Q37 -333 60 -307T108 -239T160 -136T204 27T221 250T204 473T160 636T108 740T60 807T35 839Q35 850 50 850H56H69Q197 743 256 566Q305 425 305 251Z"></path><path stroke-width="0" id="E283-MJMATHI-54" d="M40 437Q21 437 21 445Q21 450 37 501T71 602L88 651Q93 669 101 677H569H659Q691 677 697 676T704 667Q704 661 687 553T668 444Q668 437 649 437Q640 437 637 437T631 442L629 445Q629 451 635 490T641 551Q641 586 628 604T573 629Q568 630 515 631Q469 631 457 630T439 622Q438 621 368 343T298 60Q298 48 386 46Q418 46 427 45T436 36Q436 31 433 22Q429 4 424 1L422 0Q419 0 415 0Q410 0 363 1T228 2Q99 2 64 0H49Q43 6 43 9T45 27Q49 40 55 46H83H94Q174 46 189 55Q190 56 191 56Q196 59 201 76T241 233Q258 301 269 344Q339 619 339 625Q339 630 310 630H279Q212 630 191 624Q146 614 121 583T67 467Q60 445 57 441T43 437H40Z"></path><path stroke-width="0" id="E283-MJSZ4-28" d="M758 -1237T758 -1240T752 -1249H736Q718 -1249 717 -1248Q711 -1245 672 -1199Q237 -706 237 251T672 1700Q697 1730 716 1749Q718 1750 735 1750H752Q758 1744 758 1741Q758 1737 740 1713T689 1644T619 1537T540 1380T463 1176Q348 802 348 251Q348 -242 441 -599T744 -1218Q758 -1237 758 -1240Z"></path><path stroke-width="0" id="E283-MJSZ4-29" d="M33 1741Q33 1750 51 1750H60H65Q73 1750 81 1743T119 1700Q554 1207 554 251Q554 -707 119 -1199Q76 -1250 66 -1250Q65 -1250 62 -1250T56 -1249Q55 -1249 53 -1249T49 -1250Q33 -1250 33 -1239Q33 -1236 50 -1214T98 -1150T163 -1052T238 -910T311 -727Q443 -335 443 251Q443 402 436 532T405 831T339 1142T224 1438T50 1716Q33 1737 33 1741Z"></path><path stroke-width="0" id="E283-MJMATHI-52" d="M230 637Q203 637 198 638T193 649Q193 676 204 682Q206 683 378 683Q550 682 564 680Q620 672 658 652T712 606T733 563T739 529Q739 484 710 445T643 385T576 351T538 338L545 333Q612 295 612 223Q612 212 607 162T602 80V71Q602 53 603 43T614 25T640 16Q668 16 686 38T712 85Q717 99 720 102T735 105Q755 105 755 93Q755 75 731 36Q693 -21 641 -21H632Q571 -21 531 4T487 82Q487 109 502 166T517 239Q517 290 474 313Q459 320 449 321T378 323H309L277 193Q244 61 244 59Q244 55 245 54T252 50T269 48T302 46H333Q339 38 339 37T336 19Q332 6 326 0H311Q275 2 180 2Q146 2 117 2T71 2T50 1Q33 1 33 10Q33 12 36 24Q41 43 46 45Q50 46 61 46H67Q94 46 127 49Q141 52 146 61Q149 65 218 339T287 628Q287 635 230 637ZM630 554Q630 586 609 608T523 636Q521 636 500 636T462 637H440Q393 637 386 627Q385 624 352 494T319 361Q319 360 388 360Q466 361 492 367Q556 377 592 426Q608 449 619 486T630 554Z"></path><path stroke-width="0" id="E283-MJMAIN-33" d="M127 463Q100 463 85 480T69 524Q69 579 117 622T233 665Q268 665 277 664Q351 652 390 611T430 522Q430 470 396 421T302 350L299 348Q299 347 308 345T337 336T375 315Q457 262 457 175Q457 96 395 37T238 -22Q158 -22 100 21T42 130Q42 158 60 175T105 193Q133 193 151 175T169 130Q169 119 166 110T159 94T148 82T136 74T126 70T118 67L114 66Q165 21 238 21Q293 21 321 74Q338 107 338 175V195Q338 290 274 322Q259 328 213 329L171 330L168 332Q166 335 166 348Q166 366 174 366Q202 366 232 371Q266 376 294 413T322 525V533Q322 590 287 612Q265 626 240 626Q208 626 181 615T143 592T132 580H135Q138 579 143 578T153 573T165 566T175 555T183 540T186 520Q186 498 172 481T127 463Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><g transform="translate(9551,-2466)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">算</text><g transform="translate(1052,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">法</text></g><use transform="scale(1.2)" xlink:href="#E283-MJMAINB-36" x="1963" y="0"></use><use transform="scale(1.2)" xlink:href="#E283-MJMAINB-2D" x="2538" y="0"></use><use transform="scale(1.2)" xlink:href="#E283-MJMAINB-37" x="2921" y="0"></use><g transform="translate(4945,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">线</text></g><g transform="translate(5961,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">性</text></g><g transform="translate(7014,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">最</text></g><g transform="translate(8067,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">小</text></g><g transform="translate(9083,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">二</text></g><g transform="translate(10136,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">乘</text></g><use transform="scale(1.2)" xlink:href="#E283-MJMAINB-51" x="9532" y="0"></use><g transform="translate(12725,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">学</text></g><g transform="translate(13778,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">习</text></g><g transform="translate(14795,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">算</text></g><g transform="translate(15847,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">法</text></g><g transform="translate(16900,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">求</text></g><g transform="translate(17953,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">解</text></g><g transform="translate(19006,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">最</text></g><g transform="translate(20059,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">优</text></g><g transform="translate(21112,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">策</text></g><g transform="translate(22165,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">略</text></g></g><g transform="translate(0,-11798)"><g transform="translate(-19,0)"><g transform="translate(0,7869)"><g><rect fill="black" stroke="none" width="1569" height="100" x="0" y="500"></rect></g></g><g transform="translate(0,-7670)"><g><rect fill="black" stroke="none" width="1569" height="100" x="0" y="-500"></rect></g></g></g><g transform="translate(1551,0)"><g transform="translate(0,7869)"><g><rect fill="black" stroke="none" width="41597" height="100" x="0" y="500"></rect></g></g><g transform="translate(0,6569)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">输</text><g transform="translate(830,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">入</text></g><g transform="translate(1661,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">：</text></g><g transform="translate(2491,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">许</text></g><g transform="translate(3322,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">多</text></g><g transform="translate(4153,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">经</text></g><g transform="translate(4983,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">验</text></g></g><g transform="translate(0,5269)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">输</text><g transform="translate(830,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">出</text></g><g transform="translate(1661,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">：</text></g><g transform="translate(2491,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">最</text></g><g transform="translate(3322,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">优</text></g><g transform="translate(4153,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">动</text></g><g transform="translate(4983,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">作</text></g><g transform="translate(5814,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">价</text></g><g transform="translate(6645,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">值</text></g><g transform="translate(7475,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">估</text></g><g transform="translate(8306,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">计</text></g><g transform="translate(9387,0)"><use xlink:href="#E283-MJMATHI-71" x="0" y="0"></use><use xlink:href="#E283-MJMAIN-28" x="460" y="0"></use><use xlink:href="#E283-MJMATHI-73" x="849" y="0"></use><use xlink:href="#E283-MJMAIN-2C" x="1318" y="0"></use><use xlink:href="#E283-MJMATHI-61" x="1762" y="0"></use><use xlink:href="#E283-MJMAIN-3B" x="2291" y="0"></use><use xlink:href="#E283-MJMAINB-77" x="2736" y="0"></use><use xlink:href="#E283-MJMAIN-29" x="3567" y="0"></use><use xlink:href="#E283-MJMAIN-2C" x="3956" y="0"></use><use xlink:href="#E283-MJMATHI-73" x="4678" y="0"></use><use xlink:href="#E283-MJMAIN-2208" x="5425" y="0"></use><use xlink:href="#E283-MJCAL-53" x="6370" y="0"></use><use xlink:href="#E283-MJMAIN-2C" x="7012" y="0"></use><use xlink:href="#E283-MJMATHI-61" x="7457" y="0"></use><use xlink:href="#E283-MJMAIN-2208" x="8263" y="0"></use><use xlink:href="#E283-MJCAL-41" x="9208" y="0"></use></g><g transform="translate(19414,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">和</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">确</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">定</text></g><g transform="translate(2741,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">性</text></g><g transform="translate(3572,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">最</text></g><g transform="translate(4403,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">优</text></g><g transform="translate(5233,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">策</text></g><g transform="translate(6064,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">略</text></g><g transform="translate(6895,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">的</text></g><g transform="translate(7725,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">估</text></g><g transform="translate(8556,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">计</text></g></g><use xlink:href="#E283-MJMATHI-3C0" x="29051" y="0"></use><g transform="translate(29624,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">。</text></g></g></g><g transform="translate(0,3919)"><use xlink:href="#E283-MJMAIN-31"></use><use xlink:href="#E283-MJMAIN-2E" x="500" y="0"></use><g transform="translate(778,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(1608,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">初</text></g><g transform="translate(2439,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">始</text></g><g transform="translate(3269,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">化</text></g><g transform="translate(4100,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(4931,0)"><use xlink:href="#E283-MJMAINB-77" x="0" y="0"></use><use xlink:href="#E283-MJMAIN-2190" x="1108" y="0"></use></g><g transform="translate(7040,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">任</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">意</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">值</text></g><g transform="translate(2741,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">，</text></g><g transform="translate(3572,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">用</text></g></g><g transform="translate(11693,0)"><use xlink:href="#E283-MJMATHI-71" x="0" y="0"></use><use xlink:href="#E283-MJMAIN-28" x="460" y="0"></use><use xlink:href="#E283-MJMATHI-73" x="849" y="0"></use><use xlink:href="#E283-MJMAIN-2C" x="1318" y="0"></use><use xlink:href="#E283-MJMATHI-61" x="1762" y="0"></use><use xlink:href="#E283-MJMAIN-3B" x="2291" y="0"></use><use xlink:href="#E283-MJMAINB-77" x="2736" y="0"></use><use xlink:href="#E283-MJMAIN-29" x="3567" y="0"></use></g><g transform="translate(15649,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">确</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">定</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">贪</text></g><g transform="translate(2741,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">心</text></g><g transform="translate(3572,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">策</text></g><g transform="translate(4403,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">略</text></g></g><use xlink:href="#E283-MJMATHI-3C0" x="21133" y="0"></use><g transform="translate(21706,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">。</text></g></g></g><g transform="translate(0,2569)"><use xlink:href="#E283-MJMAIN-32"></use><use xlink:href="#E283-MJMAIN-2E" x="500" y="0"></use><g transform="translate(778,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(1608,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">迭</text></g><g transform="translate(2439,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">代</text></g><g transform="translate(3269,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">更</text></g><g transform="translate(4100,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">新</text></g><g transform="translate(4931,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(5761,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">迭</text></g><g transform="translate(6592,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">代</text></g><g transform="translate(7423,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">进</text></g><g transform="translate(8253,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">行</text></g><g transform="translate(9084,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">以</text></g><g transform="translate(9915,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">下</text></g><g transform="translate(10745,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">操</text></g><g transform="translate(11576,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">作</text></g><g transform="translate(12407,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">：</text></g></g><g transform="translate(0,121)"><g transform="translate(2000,0)"><use xlink:href="#E283-MJMAIN-32"></use><use xlink:href="#E283-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E283-MJMAIN-31" x="778" y="0"></use><g transform="translate(1278,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(2108,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">更</text></g><g transform="translate(2939,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">新</text></g><g transform="translate(3769,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">价</text></g><g transform="translate(4600,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">值</text></g><g transform="translate(5431,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(6261,0)"><use xlink:href="#E283-MJMAINB-77" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E283-MJMAIN-2032" x="1175" y="583"></use><use xlink:href="#E283-MJMAIN-2190" x="1403" y="0"></use><g transform="translate(2681,0)"><use xlink:href="#E283-MJSZ4-28"></use><g transform="translate(792,0)"><use xlink:href="#E283-MJSZ2-2211" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E283-MJMATHI-74" x="840" y="-1499"></use></g><use xlink:href="#E283-MJMAINB-78" x="2402" y="0"></use><use xlink:href="#E283-MJMAIN-28" x="3009" y="0"></use><g transform="translate(3398,0)"><use xlink:href="#E283-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E283-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E283-MJMAIN-2C" x="4366" y="0"></use><g transform="translate(4811,0)"><use xlink:href="#E283-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E283-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E283-MJMAIN-29" x="5916" y="0"></use><g transform="translate(6305,0)"><use xlink:href="#E283-MJSZ1-28"></use><use xlink:href="#E283-MJMAINB-78" x="458" y="0"></use><use xlink:href="#E283-MJMAIN-28" x="1065" y="0"></use><g transform="translate(1454,0)"><use xlink:href="#E283-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E283-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E283-MJMAIN-2C" x="2422" y="0"></use><g transform="translate(2866,0)"><use xlink:href="#E283-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E283-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E283-MJMAIN-29" x="3972" y="0"></use><use xlink:href="#E283-MJMAIN-2212" x="4583" y="0"></use><use xlink:href="#E283-MJMATHI-3B3" x="5583" y="0"></use><use xlink:href="#E283-MJMAINB-78" x="6126" y="0"></use><use xlink:href="#E283-MJMAIN-28" x="6733" y="0"></use><g transform="translate(7122,0)"><use xlink:href="#E283-MJMATHI-53" x="0" y="0"></use><g transform="translate(613,-150)"><use transform="scale(0.707)" xlink:href="#E283-MJMATHI-74" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E283-MJMAIN-2B" x="361" y="0"></use><use transform="scale(0.707)" xlink:href="#E283-MJMAIN-31" x="1139" y="0"></use></g></g><use xlink:href="#E283-MJMAIN-2C" x="8994" y="0"></use><g transform="translate(9439,0)"><use xlink:href="#E283-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E283-MJMAIN-2217" x="1060" y="452"></use><g transform="translate(750,-307)"><use transform="scale(0.707)" xlink:href="#E283-MJMATHI-74" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E283-MJMAIN-2B" x="361" y="0"></use><use transform="scale(0.707)" xlink:href="#E283-MJMAIN-31" x="1139" y="0"></use></g></g><use xlink:href="#E283-MJMAIN-29" x="11448" y="0"></use><use xlink:href="#E283-MJSZ1-29" x="11837" y="-1"></use><use transform="scale(0.707)" xlink:href="#E283-MJMATHI-54" x="17388" y="815"></use></g><use xlink:href="#E283-MJSZ4-29" x="19198" y="0"></use><g transform="translate(19990,1476)"><use transform="scale(0.707)" xlink:href="#E283-MJMAIN-2212" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E283-MJMAIN-31" x="778" y="0"></use></g></g><g transform="translate(23842,0)"><use xlink:href="#E283-MJSZ2-2211" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E283-MJMATHI-74" x="840" y="-1499"></use></g><g transform="translate(25452,0)"><use xlink:href="#E283-MJMATHI-52" x="0" y="0"></use><g transform="translate(759,-150)"><use transform="scale(0.707)" xlink:href="#E283-MJMATHI-74" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E283-MJMAIN-2B" x="361" y="0"></use><use transform="scale(0.707)" xlink:href="#E283-MJMAIN-31" x="1139" y="0"></use></g></g><use xlink:href="#E283-MJMAINB-78" x="27470" y="0"></use><use xlink:href="#E283-MJMAIN-28" x="28077" y="0"></use><g transform="translate(28466,0)"><use xlink:href="#E283-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E283-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E283-MJMAIN-2C" x="29435" y="0"></use><g transform="translate(29879,0)"><use xlink:href="#E283-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E283-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E283-MJMAIN-29" x="30985" y="0"></use></g><g transform="translate(37635,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">，</text></g></g></g></g><g transform="translate(0,-2229)"><g transform="translate(4444,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">其</text><g transform="translate(830,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">中</text></g><g transform="translate(1911,0)"><use xlink:href="#E283-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E283-MJMAIN-2217" x="1060" y="452"></use><g transform="translate(750,-307)"><use transform="scale(0.707)" xlink:href="#E283-MJMATHI-74" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E283-MJMAIN-2B" x="361" y="0"></use><use transform="scale(0.707)" xlink:href="#E283-MJMAIN-31" x="1139" y="0"></use></g></g><g transform="translate(3920,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">是</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">由</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">确</text></g><g transform="translate(2741,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">定</text></g><g transform="translate(3572,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">性</text></g><g transform="translate(4403,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">策</text></g><g transform="translate(5233,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">略</text></g></g><use xlink:href="#E283-MJMATHI-3C0" x="10234" y="0"></use><g transform="translate(10807,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">决</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">定</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">的</text></g><g transform="translate(2741,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">在</text></g><g transform="translate(3572,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">状</text></g><g transform="translate(4403,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">态</text></g></g><g transform="translate(16291,0)"><use xlink:href="#E283-MJMATHI-53" x="0" y="0"></use><g transform="translate(613,-150)"><use transform="scale(0.707)" xlink:href="#E283-MJMATHI-74" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E283-MJMAIN-2B" x="361" y="0"></use><use transform="scale(0.707)" xlink:href="#E283-MJMAIN-31" x="1139" y="0"></use></g></g><g transform="translate(18163,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">的</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">动</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">作</text></g><g transform="translate(2741,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">。</text></g></g></g></g><g transform="translate(0,-3702)"><g transform="translate(2000,0)"><use xlink:href="#E283-MJMAIN-32"></use><use xlink:href="#E283-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E283-MJMAIN-32" x="778" y="0"></use><g transform="translate(1278,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(2108,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">策</text></g><g transform="translate(2939,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">略</text></g><g transform="translate(3769,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">改</text></g><g transform="translate(4600,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">进</text></g><g transform="translate(5431,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(6261,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">根</text></g><g transform="translate(7092,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">据</text></g><g transform="translate(8173,0)"><use xlink:href="#E283-MJMATHI-71" x="0" y="0"></use><use xlink:href="#E283-MJMAIN-28" x="460" y="0"></use><use xlink:href="#E283-MJMATHI-73" x="849" y="0"></use><use xlink:href="#E283-MJMAIN-2C" x="1318" y="0"></use><use xlink:href="#E283-MJMATHI-61" x="1762" y="0"></use><use xlink:href="#E283-MJMAIN-3B" x="2291" y="0"></use><g transform="translate(2736,0)"><use xlink:href="#E283-MJMAINB-77" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E283-MJMAIN-2032" x="1175" y="583"></use></g><use xlink:href="#E283-MJMAIN-29" x="3861" y="0"></use></g><g transform="translate(12423,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">决</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">定</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">策</text></g><g transform="translate(2741,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">略</text></g></g><g transform="translate(16246,0)"><use xlink:href="#E283-MJMATHI-3C0" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E283-MJMAIN-2032" x="811" y="583"></use></g><g transform="translate(17115,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">。</text></g></g></g></g><g transform="translate(0,-5061)"><g transform="translate(2000,0)"><use xlink:href="#E283-MJMAIN-32"></use><use xlink:href="#E283-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E283-MJMAIN-33" x="778" y="0"></use><g transform="translate(1972,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">如</text><g transform="translate(830,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">果</text></g><g transform="translate(1661,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">达</text></g><g transform="translate(2491,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">到</text></g><g transform="translate(3322,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">迭</text></g><g transform="translate(4153,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">代</text></g><g transform="translate(4983,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">终</text></g><g transform="translate(5814,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">止</text></g><g transform="translate(6645,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">条</text></g><g transform="translate(7475,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">件</text></g><g transform="translate(8306,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(9137,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">如</text></g></g><use xlink:href="#E283-MJMAINB-77" x="12190" y="0"></use><g transform="translate(13021,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">和</text></g></g><g transform="translate(14351,0)"><use xlink:href="#E283-MJMAINB-77" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E283-MJMAIN-2032" x="1175" y="583"></use></g><g transform="translate(15477,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">足</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">够</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">接</text></g><g transform="translate(2741,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">近</text></g><g transform="translate(3572,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">，</text></g><g transform="translate(4403,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">或</text></g></g><use xlink:href="#E283-MJMATHI-3C0" x="20961" y="0"></use><g transform="translate(21534,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">和</text></g></g><g transform="translate(22864,0)"><use xlink:href="#E283-MJMATHI-3C0" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E283-MJMAIN-2032" x="811" y="583"></use></g><g transform="translate(23733,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">足</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">够</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">接</text></g><g transform="translate(2741,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">近</text></g><g transform="translate(3572,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(4403,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">，</text></g><g transform="translate(5233,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">则</text></g><g transform="translate(6064,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">终</text></g><g transform="translate(6895,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">止</text></g><g transform="translate(7725,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">迭</text></g><g transform="translate(8556,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">代</text></g><g transform="translate(9387,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">；</text></g></g></g></g><g transform="translate(0,-6370)"><g transform="translate(4000,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">否</text><g transform="translate(830,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">则</text></g><g transform="translate(1661,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">更</text></g><g transform="translate(2491,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">新</text></g><g transform="translate(3572,0)"><use xlink:href="#E283-MJMAINB-77" x="0" y="0"></use><use xlink:href="#E283-MJMAIN-2190" x="1108" y="0"></use><g transform="translate(2386,0)"><use xlink:href="#E283-MJMAINB-77" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E283-MJMAIN-2032" x="1175" y="583"></use></g></g><g transform="translate(7084,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">，</text></g></g><g transform="translate(8165,0)"><use xlink:href="#E283-MJMATHI-3C0" x="0" y="0"></use><use xlink:href="#E283-MJMAIN-2190" x="850" y="0"></use><g transform="translate(2128,0)"><use xlink:href="#E283-MJMATHI-3C0" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E283-MJMAIN-2032" x="811" y="583"></use></g></g><g transform="translate(11162,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">进</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">行</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">下</text></g><g transform="translate(2741,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">一</text></g><g transform="translate(3572,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">轮</text></g><g transform="translate(4403,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">迭</text></g><g transform="translate(5233,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">代</text></g><g transform="translate(6064,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">。</text></g></g></g></g><g transform="translate(0,-7670)"><g><rect fill="black" stroke="none" width="41597" height="100" x="0" y="-500"></rect></g></g></g></g></g></svg></span></div><script type="math/tex; mode=display" id="MathJax-Element-256">\; \\ \; \\
\large \textbf{算法 6-7   线性最小二乘 Q 学习算法求解最优策略} \\
\begin{split}
\rule[5pt]{10mm}{0.1em} &\rule[5pt]{265mm}{0.1em} \\
&\text{输入：许多经验} \\
&\text{输出：最优动作价值估计 $q(s,a;\bold w),\; s \in \mathcal S,a \in \mathcal A$ 和确定性最优策略的估计 $\pi$ 。} \\
&\text{1.（初始化）$\bold w \leftarrow$ 任意值，用 $q(s,a;\bold w)$ 确定贪心策略 $\pi$ 。} \\
&\text{2.（迭代更新）迭代进行以下操作：} \\
&\qquad \text{2.1（更新价值）$\bold w' \leftarrow \left(\sum_t \bold x(S_t,A_t)\left(\bold x(S_t,A_t) - \gamma \bold x(S_{t+1},A_{t+1}^*)\right)^T\right)^{-1} \sum_t R_{t+1}\bold x(S_t,A_t)$ ，} \\
&\qquad \qquad \;\, \text{其中 $A_{t+1}^*$ 是由确定性策略 $\pi$ 决定的在状态 $S_{t+1}$ 的动作。}\\
&\qquad \text{2.2（策略改进）根据 $q(s,a;\bold w')$ 决定策略 $\pi'$ 。} \\
&\qquad \text{2.3 $\;\,$如果达到迭代终止条件（如 $\bold w$ 和 $\bold w'$ 足够接近，或 $\pi$ 和 $\pi'$ 足够接近），则终止迭代；}\\
&\qquad \qquad \text{否则更新 $\bold w \leftarrow \bold w'$ ，$\pi \leftarrow \pi'$ 进行下一轮迭代。} \\
\rule[-5pt]{10mm}{0.1em} &\rule[-5pt]{265mm}{0.1em}
\end{split}
\; \\ \; \\</script></div></div><h3><a name="三函数近似的收敛性" class="md-header-anchor"></a><span>三、函数近似的收敛性</span></h3><p><span>线性近似具有简单的线性叠加结构，这使得线性近似可以获得额外的收敛性；对于函数近似算法，收敛性往往只在采用梯度下降的回合更新时有保证，而在采用半梯度下降的时序差分方法时是没有保证的。各种收敛情况在下列表中给出，其中查表法是指不采用函数近似的方法；所有的收敛性都是在学习率满足 Robbins-Monro 序列下才具有的，且一般都可以通过验证随机近似  Robbins-Monro 算法的条件证明，对于最优策略求解的收敛性证明，则需要用到了其随机优化的版本。</span></p><table border="2" frame="hsides">
    <caption><b>策略评估算法的收敛性</b></caption>
    <tbody><tr align="center">
        <th colspan="2">学习方法</th>
        <th>查表法</th>
        <th>线性近似</th>
        <th>非线性近似</th>
    </tr>
    <tr align="center">
        <td rowspan="4" style="vertical-align: middle">同策</td>
        <td>回合更新</td>
        <td>收敛</td>
        <td>收敛</td>
        <td>收敛</td>
    </tr>
    <tr align="center">
        <td>线性最小二乘回合更新</td>
        <td>收敛</td>
        <td>收敛</td>
        <td>不适用</td>
    </tr>
    <tr align="center">
        <td>时序差分更新</td>
        <td>收敛</td>
        <td>收敛</td>
        <td>不一定收敛</td>
    </tr>
    <tr align="center">
        <td>线性最小二乘时序差分更新</td>
        <td>收敛</td>
        <td>收敛</td>
        <td>不适用</td>
    </tr>
    <tr align="center">
        <td rowspan="4" style="vertical-align: middle">异策</td>
        <td>回合更新</td>
        <td>收敛</td>
        <td>收敛</td>
        <td>收敛</td>
    </tr>
    <tr align="center">
        <td>线性最小二乘回合更新</td>
        <td>收敛</td>
        <td>收敛</td>
        <td>不适用</td>
    </tr>
    <tr align="center">
        <td>时序差分更新</td>
        <td>收敛</td>
        <td>不一定收敛</td>
        <td>不一定收敛</td>
    </tr>
    <tr align="center">
        <td>线性最小二乘时序差分更新</td>
        <td>收敛</td>
        <td>收敛</td>
        <td>不适用</td>
    </tr>
</tbody></table>
<table border="2" frame="hsides">
    <caption> <br><b>最优策略求解算法的收敛性</b></caption>
    <tbody><tr align="center">
        <th>学习方法</th>
        <th>查表法</th>
        <th>线性近似</th>
        <th>非线性近似</th>
    </tr>
    <tr align="center">
        <td>回合更新</td>
        <td>收敛</td>
        <td>收敛或在最优解附近摆动</td>
        <td>不一定收敛</td>
    </tr>
    <tr align="center">
        <td>SARSA</td>
        <td>收敛</td>
        <td>收敛或在最优解附近摆动</td>
        <td>不一定收敛</td>
    </tr>
    <tr align="center">
        <td>Q 学习</td>
        <td>收敛</td>
        <td>不一定收敛</td>
        <td>不一定收敛</td>
    </tr>
    <tr align="center">
        <td>最小二乘迭代更新</td>
        <td>收敛</td>
        <td>收敛或在最优解附近摆动</td>
        <td>不适用</td>
    </tr>    
</tbody></table><p><span>值得一提的是，对于异策 Q 学习，即使采用了线性近似，仍然不能保证收敛。研究人员发现，只要异策、自益、函数近似这三者同时出现，就不能保证收敛性，但有一个著名的反例叫做 Baird 反例（Baird&#39;s counterexample）。</span></p><h3><a name="四深度-q-学习" class="md-header-anchor"></a><span>四、深度 Q 学习</span></h3><p><span>深度 Q 学习是目前非常热门的函数近似方法，为了解决无法保证收敛性而导致的训练不稳定或训练困难的问题，研究人员主要从以下两个方面进行了改进：</span></p><ul><li><strong><span>经验回放</span></strong><span>（experience replay）：将经验（即历史的状态、动作、奖励等）存储起来，再按一定规则采样存储的经验。</span></li><li><strong><span>目标网络</span></strong><span>（target network）：修改网络的更新方式，例如不把刚学习的网络权重马上用于后续的自益过程。</span></li></ul><p><span>V. Mnih 等在 2013 年发表的《Playing Atari with deep reinforcement learning》提出了基于经验回放的深度 Q 网络，标志着深度 Q 网络的诞生，也标志着深度强化学习的诞生。经验回放就是一种让经验的概率分布变得稳定的技术，它能提高训练的稳定性，其主要步骤为：</span></p><ol start='' ><li><span>存储：将轨迹以 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="18.756ex" height="2.71ex" viewBox="0 -832.7 8075.4 1166.9" role="img" focusable="false" style="vertical-align: -0.776ex;"><defs><path stroke-width="0" id="E322-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E322-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E322-MJMATHI-74" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path><path stroke-width="0" id="E322-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E322-MJMATHI-41" d="M208 74Q208 50 254 46Q272 46 272 35Q272 34 270 22Q267 8 264 4T251 0Q249 0 239 0T205 1T141 2Q70 2 50 0H42Q35 7 35 11Q37 38 48 46H62Q132 49 164 96Q170 102 345 401T523 704Q530 716 547 716H555H572Q578 707 578 706L606 383Q634 60 636 57Q641 46 701 46Q726 46 726 36Q726 34 723 22Q720 7 718 4T704 0Q701 0 690 0T651 1T578 2Q484 2 455 0H443Q437 6 437 9T439 27Q443 40 445 43L449 46H469Q523 49 533 63L521 213H283L249 155Q208 86 208 74ZM516 260Q516 271 504 416T490 562L463 519Q447 492 400 412L310 260L413 259Q516 259 516 260Z"></path><path stroke-width="0" id="E322-MJMATHI-52" d="M230 637Q203 637 198 638T193 649Q193 676 204 682Q206 683 378 683Q550 682 564 680Q620 672 658 652T712 606T733 563T739 529Q739 484 710 445T643 385T576 351T538 338L545 333Q612 295 612 223Q612 212 607 162T602 80V71Q602 53 603 43T614 25T640 16Q668 16 686 38T712 85Q717 99 720 102T735 105Q755 105 755 93Q755 75 731 36Q693 -21 641 -21H632Q571 -21 531 4T487 82Q487 109 502 166T517 239Q517 290 474 313Q459 320 449 321T378 323H309L277 193Q244 61 244 59Q244 55 245 54T252 50T269 48T302 46H333Q339 38 339 37T336 19Q332 6 326 0H311Q275 2 180 2Q146 2 117 2T71 2T50 1Q33 1 33 10Q33 12 36 24Q41 43 46 45Q50 46 61 46H67Q94 46 127 49Q141 52 146 61Q149 65 218 339T287 628Q287 635 230 637ZM630 554Q630 586 609 608T523 636Q521 636 500 636T462 637H440Q393 637 386 627Q385 624 352 494T319 361Q319 360 388 360Q466 361 492 367Q556 377 592 426Q608 449 619 486T630 554Z"></path><path stroke-width="0" id="E322-MJMAIN-2B" d="M56 237T56 250T70 270H369V420L370 570Q380 583 389 583Q402 583 409 568V270H707Q722 262 722 250T707 230H409V-68Q401 -82 391 -82H389H387Q375 -82 369 -68V230H70Q56 237 56 250Z"></path><path stroke-width="0" id="E322-MJMAIN-31" d="M213 578L200 573Q186 568 160 563T102 556H83V602H102Q149 604 189 617T245 641T273 663Q275 666 285 666Q294 666 302 660V361L303 61Q310 54 315 52T339 48T401 46H427V0H416Q395 3 257 3Q121 3 100 0H88V46H114Q136 46 152 46T177 47T193 50T201 52T207 57T213 61V578Z"></path><path stroke-width="0" id="E322-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E322-MJMAIN-28" x="0" y="0"></use><g transform="translate(389,0)"><use xlink:href="#E322-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E322-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E322-MJMAIN-2C" x="1357" y="0"></use><g transform="translate(1801,0)"><use xlink:href="#E322-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E322-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E322-MJMAIN-2C" x="2907" y="0"></use><g transform="translate(3351,0)"><use xlink:href="#E322-MJMATHI-52" x="0" y="0"></use><g transform="translate(759,-150)"><use transform="scale(0.707)" xlink:href="#E322-MJMATHI-74" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E322-MJMAIN-2B" x="361" y="0"></use><use transform="scale(0.707)" xlink:href="#E322-MJMAIN-31" x="1139" y="0"></use></g></g><use xlink:href="#E322-MJMAIN-2C" x="5369" y="0"></use><g transform="translate(5814,0)"><use xlink:href="#E322-MJMATHI-53" x="0" y="0"></use><g transform="translate(613,-150)"><use transform="scale(0.707)" xlink:href="#E322-MJMATHI-74" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E322-MJMAIN-2B" x="361" y="0"></use><use transform="scale(0.707)" xlink:href="#E322-MJMAIN-31" x="1139" y="0"></use></g></g><use xlink:href="#E322-MJMAIN-29" x="7686" y="0"></use></g></svg></span><script type="math/tex">(S_t,A_t,R_{t+1},S_{t+1})</script><span> 等形式存储起来；</span></li><li><span>采样回放：使用某种规则从存储的  </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="18.756ex" height="2.71ex" viewBox="0 -832.7 8075.4 1166.9" role="img" focusable="false" style="vertical-align: -0.776ex;"><defs><path stroke-width="0" id="E322-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E322-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E322-MJMATHI-74" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path><path stroke-width="0" id="E322-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E322-MJMATHI-41" d="M208 74Q208 50 254 46Q272 46 272 35Q272 34 270 22Q267 8 264 4T251 0Q249 0 239 0T205 1T141 2Q70 2 50 0H42Q35 7 35 11Q37 38 48 46H62Q132 49 164 96Q170 102 345 401T523 704Q530 716 547 716H555H572Q578 707 578 706L606 383Q634 60 636 57Q641 46 701 46Q726 46 726 36Q726 34 723 22Q720 7 718 4T704 0Q701 0 690 0T651 1T578 2Q484 2 455 0H443Q437 6 437 9T439 27Q443 40 445 43L449 46H469Q523 49 533 63L521 213H283L249 155Q208 86 208 74ZM516 260Q516 271 504 416T490 562L463 519Q447 492 400 412L310 260L413 259Q516 259 516 260Z"></path><path stroke-width="0" id="E322-MJMATHI-52" d="M230 637Q203 637 198 638T193 649Q193 676 204 682Q206 683 378 683Q550 682 564 680Q620 672 658 652T712 606T733 563T739 529Q739 484 710 445T643 385T576 351T538 338L545 333Q612 295 612 223Q612 212 607 162T602 80V71Q602 53 603 43T614 25T640 16Q668 16 686 38T712 85Q717 99 720 102T735 105Q755 105 755 93Q755 75 731 36Q693 -21 641 -21H632Q571 -21 531 4T487 82Q487 109 502 166T517 239Q517 290 474 313Q459 320 449 321T378 323H309L277 193Q244 61 244 59Q244 55 245 54T252 50T269 48T302 46H333Q339 38 339 37T336 19Q332 6 326 0H311Q275 2 180 2Q146 2 117 2T71 2T50 1Q33 1 33 10Q33 12 36 24Q41 43 46 45Q50 46 61 46H67Q94 46 127 49Q141 52 146 61Q149 65 218 339T287 628Q287 635 230 637ZM630 554Q630 586 609 608T523 636Q521 636 500 636T462 637H440Q393 637 386 627Q385 624 352 494T319 361Q319 360 388 360Q466 361 492 367Q556 377 592 426Q608 449 619 486T630 554Z"></path><path stroke-width="0" id="E322-MJMAIN-2B" d="M56 237T56 250T70 270H369V420L370 570Q380 583 389 583Q402 583 409 568V270H707Q722 262 722 250T707 230H409V-68Q401 -82 391 -82H389H387Q375 -82 369 -68V230H70Q56 237 56 250Z"></path><path stroke-width="0" id="E322-MJMAIN-31" d="M213 578L200 573Q186 568 160 563T102 556H83V602H102Q149 604 189 617T245 641T273 663Q275 666 285 666Q294 666 302 660V361L303 61Q310 54 315 52T339 48T401 46H427V0H416Q395 3 257 3Q121 3 100 0H88V46H114Q136 46 152 46T177 47T193 50T201 52T207 57T213 61V578Z"></path><path stroke-width="0" id="E322-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E322-MJMAIN-28" x="0" y="0"></use><g transform="translate(389,0)"><use xlink:href="#E322-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E322-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E322-MJMAIN-2C" x="1357" y="0"></use><g transform="translate(1801,0)"><use xlink:href="#E322-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E322-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E322-MJMAIN-2C" x="2907" y="0"></use><g transform="translate(3351,0)"><use xlink:href="#E322-MJMATHI-52" x="0" y="0"></use><g transform="translate(759,-150)"><use transform="scale(0.707)" xlink:href="#E322-MJMATHI-74" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E322-MJMAIN-2B" x="361" y="0"></use><use transform="scale(0.707)" xlink:href="#E322-MJMAIN-31" x="1139" y="0"></use></g></g><use xlink:href="#E322-MJMAIN-2C" x="5369" y="0"></use><g transform="translate(5814,0)"><use xlink:href="#E322-MJMATHI-53" x="0" y="0"></use><g transform="translate(613,-150)"><use transform="scale(0.707)" xlink:href="#E322-MJMATHI-74" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E322-MJMAIN-2B" x="361" y="0"></use><use transform="scale(0.707)" xlink:href="#E322-MJMAIN-31" x="1139" y="0"></use></g></g><use xlink:href="#E322-MJMAIN-29" x="7686" y="0"></use></g></svg></span><script type="math/tex">(S_t,A_t,R_{t+1},S_{t+1})</script><span> 中随机取出一条或多条经验。</span></li></ol><p><span>下面给出了带经验回放的 Q 学习最优策略求解算法：</span></p><div contenteditable="false" spellcheck="false" class="mathjax-block md-end-block md-math-block md-rawblock" id="mathjax-n67" cid="n67" mdtype="math_block"><div class="md-rawblock-container md-math-container" tabindex="-1"><div class="MathJax_SVG_Display"><span class="MathJax_SVG" id="MathJax-Element-257-Frame" tabindex="-1" style="font-size: 100%; display: inline-block; zoom: 0.946724;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="103.927ex" height="49.591ex" viewBox="-18.1 -43.5 44746.2 21351.7" role="img" focusable="false" style="vertical-align: -49.49ex; margin-left: -0.042ex; max-width: 100%;"><defs><path stroke-width="0" id="E284-MJMAINB-36" d="M48 318Q48 395 68 456T120 553T193 613T273 646T350 655Q425 655 461 616T497 524Q497 485 475 468T428 451Q399 451 378 470T357 521Q357 565 403 588Q375 601 351 601Q313 601 282 584Q242 565 222 526Q199 473 199 367Q201 369 210 380T227 396T246 410T275 422T312 426Q438 426 494 332Q526 285 526 208V199Q526 112 465 53Q428 17 388 3T285 -11Q236 -11 195 7T135 43T104 80Q48 165 48 318ZM375 231V244V268Q375 295 373 310T364 342T341 366T299 374H297Q231 374 208 287Q200 257 200 196Q201 120 209 100Q231 47 288 47Q351 47 368 90Q375 112 375 231Z"></path><path stroke-width="0" id="E284-MJMAINB-2D" d="M13 166V278H318V166H13Z"></path><path stroke-width="0" id="E284-MJMAINB-38" d="M80 474Q80 561 139 607T278 654Q357 654 411 632Q490 593 494 509Q494 424 416 376L407 371L418 364Q432 356 447 345T481 312T513 260T526 192Q526 100 461 45T285 -11Q184 -11 116 32T48 164Q48 181 50 196T58 225T69 249T84 270T100 286T117 300T134 311T149 321T162 329L152 336Q120 360 100 397T80 474ZM347 404Q404 446 404 503Q404 579 317 599Q309 600 276 600Q178 600 170 538Q170 532 171 527T173 518T178 509T184 501T194 492T205 484T219 476T235 467T254 456T275 445L347 404ZM289 47Q323 47 351 54T402 82T425 137Q425 147 421 161Q411 183 391 197T303 249Q224 293 223 293Q220 291 215 288T197 273T175 248T157 213T149 167Q149 109 188 78T289 47Z"></path><path stroke-width="0" id="E284-MJMAINB-51" d="M64 339Q64 431 96 502T182 614T295 675T420 696Q469 696 481 695Q620 680 709 589T798 339Q798 255 768 184Q720 77 611 26L600 21Q635 -26 682 -26H696Q769 -26 769 0Q769 7 774 12T787 18Q805 18 805 -7V-13Q803 -64 785 -106T737 -171Q720 -183 697 -191Q687 -193 668 -193Q636 -193 613 -182T575 -144T552 -94T532 -27Q531 -23 530 -16T528 -6T526 -3L512 -5Q499 -7 477 -8T431 -10Q393 -10 382 -9Q238 8 151 97T64 339ZM326 80Q326 113 356 138T430 163Q492 163 542 100L553 86Q554 85 561 91T578 108Q637 179 637 330Q637 430 619 498T548 604Q500 641 425 641Q408 641 390 637T347 623T299 590T259 535Q226 469 226 338Q226 244 246 180T318 79L325 74Q326 74 326 80ZM506 58Q480 112 433 112Q412 112 395 104T378 77Q378 44 431 44Q480 44 506 58Z"></path><path stroke-width="0" id="E284-MJMAIN-31" d="M213 578L200 573Q186 568 160 563T102 556H83V602H102Q149 604 189 617T245 641T273 663Q275 666 285 666Q294 666 302 660V361L303 61Q310 54 315 52T339 48T401 46H427V0H416Q395 3 257 3Q121 3 100 0H88V46H114Q136 46 152 46T177 47T193 50T201 52T207 57T213 61V578Z"></path><path stroke-width="0" id="E284-MJMAIN-2E" d="M78 60Q78 84 95 102T138 120Q162 120 180 104T199 61Q199 36 182 18T139 0T96 17T78 60Z"></path><path stroke-width="0" id="E284-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E284-MJMAIN-32" d="M109 429Q82 429 66 447T50 491Q50 562 103 614T235 666Q326 666 387 610T449 465Q449 422 429 383T381 315T301 241Q265 210 201 149L142 93L218 92Q375 92 385 97Q392 99 409 186V189H449V186Q448 183 436 95T421 3V0H50V19V31Q50 38 56 46T86 81Q115 113 136 137Q145 147 170 174T204 211T233 244T261 278T284 308T305 340T320 369T333 401T340 431T343 464Q343 527 309 573T212 619Q179 619 154 602T119 569T109 550Q109 549 114 549Q132 549 151 535T170 489Q170 464 154 447T109 429Z"></path><path stroke-width="0" id="E284-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E284-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E284-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E284-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E284-MJMAIN-22C5" d="M78 250Q78 274 95 292T138 310Q162 310 180 294T199 251Q199 226 182 208T139 190T96 207T78 250Z"></path><path stroke-width="0" id="E284-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E284-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E284-MJMATHI-41" d="M208 74Q208 50 254 46Q272 46 272 35Q272 34 270 22Q267 8 264 4T251 0Q249 0 239 0T205 1T141 2Q70 2 50 0H42Q35 7 35 11Q37 38 48 46H62Q132 49 164 96Q170 102 345 401T523 704Q530 716 547 716H555H572Q578 707 578 706L606 383Q634 60 636 57Q641 46 701 46Q726 46 726 36Q726 34 723 22Q720 7 718 4T704 0Q701 0 690 0T651 1T578 2Q484 2 455 0H443Q437 6 437 9T439 27Q443 40 445 43L449 46H469Q523 49 533 63L521 213H283L249 155Q208 86 208 74ZM516 260Q516 271 504 416T490 562L463 519Q447 492 400 412L310 260L413 259Q516 259 516 260Z"></path><path stroke-width="0" id="E284-MJMATHI-52" d="M230 637Q203 637 198 638T193 649Q193 676 204 682Q206 683 378 683Q550 682 564 680Q620 672 658 652T712 606T733 563T739 529Q739 484 710 445T643 385T576 351T538 338L545 333Q612 295 612 223Q612 212 607 162T602 80V71Q602 53 603 43T614 25T640 16Q668 16 686 38T712 85Q717 99 720 102T735 105Q755 105 755 93Q755 75 731 36Q693 -21 641 -21H632Q571 -21 531 4T487 82Q487 109 502 166T517 239Q517 290 474 313Q459 320 449 321T378 323H309L277 193Q244 61 244 59Q244 55 245 54T252 50T269 48T302 46H333Q339 38 339 37T336 19Q332 6 326 0H311Q275 2 180 2Q146 2 117 2T71 2T50 1Q33 1 33 10Q33 12 36 24Q41 43 46 45Q50 46 61 46H67Q94 46 127 49Q141 52 146 61Q149 65 218 339T287 628Q287 635 230 637ZM630 554Q630 586 609 608T523 636Q521 636 500 636T462 637H440Q393 637 386 627Q385 624 352 494T319 361Q319 360 388 360Q466 361 492 367Q556 377 592 426Q608 449 619 486T630 554Z"></path><path stroke-width="0" id="E284-MJMAIN-2032" d="M79 43Q73 43 52 49T30 61Q30 68 85 293T146 528Q161 560 198 560Q218 560 240 545T262 501Q262 496 260 486Q259 479 173 263T84 45T79 43Z"></path><path stroke-width="0" id="E284-MJMAIN-33" d="M127 463Q100 463 85 480T69 524Q69 579 117 622T233 665Q268 665 277 664Q351 652 390 611T430 522Q430 470 396 421T302 350L299 348Q299 347 308 345T337 336T375 315Q457 262 457 175Q457 96 395 37T238 -22Q158 -22 100 21T42 130Q42 158 60 175T105 193Q133 193 151 175T169 130Q169 119 166 110T159 94T148 82T136 74T126 70T118 67L114 66Q165 21 238 21Q293 21 321 74Q338 107 338 175V195Q338 290 274 322Q259 328 213 329L171 330L168 332Q166 335 166 348Q166 366 174 366Q202 366 232 371Q266 376 294 413T322 525V533Q322 590 287 612Q265 626 240 626Q208 626 181 615T143 592T132 580H135Q138 579 143 578T153 573T165 566T175 555T183 540T186 520Q186 498 172 481T127 463Z"></path><path stroke-width="0" id="E284-MJMATHI-69" d="M184 600Q184 624 203 642T247 661Q265 661 277 649T290 619Q290 596 270 577T226 557Q211 557 198 567T184 600ZM21 287Q21 295 30 318T54 369T98 420T158 442Q197 442 223 419T250 357Q250 340 236 301T196 196T154 83Q149 61 149 51Q149 26 166 26Q175 26 185 29T208 43T235 78T260 137Q263 149 265 151T282 153Q302 153 302 143Q302 135 293 112T268 61T223 11T161 -11Q129 -11 102 10T74 74Q74 91 79 106T122 220Q160 321 166 341T173 380Q173 404 156 404H154Q124 404 99 371T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Z"></path><path stroke-width="0" id="E284-MJMAIN-34" d="M462 0Q444 3 333 3Q217 3 199 0H190V46H221Q241 46 248 46T265 48T279 53T286 61Q287 63 287 115V165H28V211L179 442Q332 674 334 675Q336 677 355 677H373L379 671V211H471V165H379V114Q379 73 379 66T385 54Q393 47 442 46H471V0H462ZM293 211V545L74 212L183 211H293Z"></path><path stroke-width="0" id="E284-MJMATHI-55" d="M107 637Q73 637 71 641Q70 643 70 649Q70 673 81 682Q83 683 98 683Q139 681 234 681Q268 681 297 681T342 682T362 682Q378 682 378 672Q378 670 376 658Q371 641 366 638H364Q362 638 359 638T352 638T343 637T334 637Q295 636 284 634T266 623Q265 621 238 518T184 302T154 169Q152 155 152 140Q152 86 183 55T269 24Q336 24 403 69T501 205L552 406Q599 598 599 606Q599 633 535 637Q511 637 511 648Q511 650 513 660Q517 676 519 679T529 683Q532 683 561 682T645 680Q696 680 723 681T752 682Q767 682 767 672Q767 650 759 642Q756 637 737 637Q666 633 648 597Q646 592 598 404Q557 235 548 205Q515 105 433 42T263 -22Q171 -22 116 34T60 167V183Q60 201 115 421Q164 622 164 628Q164 635 107 637Z"></path><path stroke-width="0" id="E284-MJMAIN-2190" d="M944 261T944 250T929 230H165Q167 228 182 216T211 189T244 152T277 96T303 25Q308 7 308 0Q308 -11 288 -11Q281 -11 278 -11T272 -7T267 2T263 21Q245 94 195 151T73 236Q58 242 55 247Q55 254 59 257T73 264Q121 283 158 314T215 375T247 434T264 480L267 497Q269 503 270 505T275 509T288 511Q308 511 308 500Q308 493 303 475Q293 438 278 406T246 352T215 315T185 287T165 270H929Q944 261 944 250Z"></path><path stroke-width="0" id="E284-MJMAIN-2B" d="M56 237T56 250T70 270H369V420L370 570Q380 583 389 583Q402 583 409 568V270H707Q722 262 722 250T707 230H409V-68Q401 -82 391 -82H389H387Q375 -82 369 -68V230H70Q56 237 56 250Z"></path><path stroke-width="0" id="E284-MJMATHI-3B3" d="M31 249Q11 249 11 258Q11 275 26 304T66 365T129 418T206 441Q233 441 239 440Q287 429 318 386T371 255Q385 195 385 170Q385 166 386 166L398 193Q418 244 443 300T486 391T508 430Q510 431 524 431H537Q543 425 543 422Q543 418 522 378T463 251T391 71Q385 55 378 6T357 -100Q341 -165 330 -190T303 -216Q286 -216 286 -188Q286 -138 340 32L346 51L347 69Q348 79 348 100Q348 257 291 317Q251 355 196 355Q148 355 108 329T51 260Q49 251 47 251Q45 249 31 249Z"></path><path stroke-width="0" id="E284-MJMAIN-6D" d="M41 46H55Q94 46 102 60V68Q102 77 102 91T102 122T103 161T103 203Q103 234 103 269T102 328V351Q99 370 88 376T43 385H25V408Q25 431 27 431L37 432Q47 433 65 434T102 436Q119 437 138 438T167 441T178 442H181V402Q181 364 182 364T187 369T199 384T218 402T247 421T285 437Q305 442 336 442Q351 442 364 440T387 434T406 426T421 417T432 406T441 395T448 384T452 374T455 366L457 361L460 365Q463 369 466 373T475 384T488 397T503 410T523 422T546 432T572 439T603 442Q729 442 740 329Q741 322 741 190V104Q741 66 743 59T754 49Q775 46 803 46H819V0H811L788 1Q764 2 737 2T699 3Q596 3 587 0H579V46H595Q656 46 656 62Q657 64 657 200Q656 335 655 343Q649 371 635 385T611 402T585 404Q540 404 506 370Q479 343 472 315T464 232V168V108Q464 78 465 68T468 55T477 49Q498 46 526 46H542V0H534L510 1Q487 2 460 2T422 3Q319 3 310 0H302V46H318Q379 46 379 62Q380 64 380 200Q379 335 378 343Q372 371 358 385T334 402T308 404Q263 404 229 370Q202 343 195 315T187 232V168V108Q187 78 188 68T191 55T200 49Q221 46 249 46H265V0H257L234 1Q210 2 183 2T145 3Q42 3 33 0H25V46H41Z"></path><path stroke-width="0" id="E284-MJMAIN-61" d="M137 305T115 305T78 320T63 359Q63 394 97 421T218 448Q291 448 336 416T396 340Q401 326 401 309T402 194V124Q402 76 407 58T428 40Q443 40 448 56T453 109V145H493V106Q492 66 490 59Q481 29 455 12T400 -6T353 12T329 54V58L327 55Q325 52 322 49T314 40T302 29T287 17T269 6T247 -2T221 -8T190 -11Q130 -11 82 20T34 107Q34 128 41 147T68 188T116 225T194 253T304 268H318V290Q318 324 312 340Q290 411 215 411Q197 411 181 410T156 406T148 403Q170 388 170 359Q170 334 154 320ZM126 106Q126 75 150 51T209 26Q247 26 276 49T315 109Q317 116 318 175Q318 233 317 233Q309 233 296 232T251 223T193 203T147 166T126 106Z"></path><path stroke-width="0" id="E284-MJMAIN-78" d="M201 0Q189 3 102 3Q26 3 17 0H11V46H25Q48 47 67 52T96 61T121 78T139 96T160 122T180 150L226 210L168 288Q159 301 149 315T133 336T122 351T113 363T107 370T100 376T94 379T88 381T80 383Q74 383 44 385H16V431H23Q59 429 126 429Q219 429 229 431H237V385Q201 381 201 369Q201 367 211 353T239 315T268 274L272 270L297 304Q329 345 329 358Q329 364 327 369T322 376T317 380T310 384L307 385H302V431H309Q324 428 408 428Q487 428 493 431H499V385H492Q443 385 411 368Q394 360 377 341T312 257L296 236L358 151Q424 61 429 57T446 50Q464 46 499 46H516V0H510H502Q494 1 482 1T457 2T432 2T414 3Q403 3 377 3T327 1L304 0H295V46H298Q309 46 320 51T331 63Q331 65 291 120L250 175Q249 174 219 133T185 88Q181 83 181 74Q181 63 188 55T206 46Q208 46 208 23V0H201Z"></path><path stroke-width="0" id="E284-MJMATHI-61" d="M33 157Q33 258 109 349T280 441Q331 441 370 392Q386 422 416 422Q429 422 439 414T449 394Q449 381 412 234T374 68Q374 43 381 35T402 26Q411 27 422 35Q443 55 463 131Q469 151 473 152Q475 153 483 153H487Q506 153 506 144Q506 138 501 117T481 63T449 13Q436 0 417 -8Q409 -10 393 -10Q359 -10 336 5T306 36L300 51Q299 52 296 50Q294 48 292 46Q233 -10 172 -10Q117 -10 75 30T33 157ZM351 328Q351 334 346 350T323 385T277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q217 26 254 59T298 110Q300 114 325 217T351 328Z"></path><path stroke-width="0" id="E284-MJMAIN-35" d="M164 157Q164 133 148 117T109 101H102Q148 22 224 22Q294 22 326 82Q345 115 345 210Q345 313 318 349Q292 382 260 382H254Q176 382 136 314Q132 307 129 306T114 304Q97 304 95 310Q93 314 93 485V614Q93 664 98 664Q100 666 102 666Q103 666 123 658T178 642T253 634Q324 634 389 662Q397 666 402 666Q410 666 410 648V635Q328 538 205 538Q174 538 149 544L139 546V374Q158 388 169 396T205 412T256 420Q337 420 393 355T449 201Q449 109 385 44T229 -22Q148 -22 99 32T50 154Q50 178 61 192T84 210T107 214Q132 214 148 197T164 157Z"></path><path stroke-width="0" id="E284-MJMAIN-5B" d="M118 -250V750H255V710H158V-210H255V-250H118Z"></path><path stroke-width="0" id="E284-MJMAIN-2212" d="M84 237T84 250T98 270H679Q694 262 694 250T679 230H98Q84 237 84 250Z"></path><path stroke-width="0" id="E284-MJMAIN-5D" d="M22 710V750H159V-250H22V-210H119V710H22Z"></path><path stroke-width="0" id="E284-MJMATHI-3B1" d="M34 156Q34 270 120 356T309 442Q379 442 421 402T478 304Q484 275 485 237V208Q534 282 560 374Q564 388 566 390T582 393Q603 393 603 385Q603 376 594 346T558 261T497 161L486 147L487 123Q489 67 495 47T514 26Q528 28 540 37T557 60Q559 67 562 68T577 70Q597 70 597 62Q597 56 591 43Q579 19 556 5T512 -10H505Q438 -10 414 62L411 69L400 61Q390 53 370 41T325 18T267 -2T203 -11Q124 -11 79 39T34 156ZM208 26Q257 26 306 47T379 90L403 112Q401 255 396 290Q382 405 304 405Q235 405 183 332Q156 292 139 224T121 120Q121 71 146 49T208 26Z"></path><path stroke-width="0" id="E284-MJMAIN-2207" d="M46 676Q46 679 51 683H781Q786 679 786 676Q786 674 617 326T444 -26Q439 -33 416 -33T388 -26Q385 -22 216 326T46 676ZM697 596Q697 597 445 597T193 596Q195 591 319 336T445 80L697 596Z"></path><path stroke-width="0" id="E284-MJMAIN-36" d="M42 313Q42 476 123 571T303 666Q372 666 402 630T432 550Q432 525 418 510T379 495Q356 495 341 509T326 548Q326 592 373 601Q351 623 311 626Q240 626 194 566Q147 500 147 364L148 360Q153 366 156 373Q197 433 263 433H267Q313 433 348 414Q372 400 396 374T435 317Q456 268 456 210V192Q456 169 451 149Q440 90 387 34T253 -22Q225 -22 199 -14T143 16T92 75T56 172T42 313ZM257 397Q227 397 205 380T171 335T154 278T148 216Q148 133 160 97T198 39Q222 21 251 21Q302 21 329 59Q342 77 347 104T352 209Q352 289 347 316T329 361Q302 397 257 397Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><g transform="translate(10604,-2466)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">算</text><g transform="translate(1052,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">法</text></g><use transform="scale(1.2)" xlink:href="#E284-MJMAINB-36" x="1963" y="0"></use><use transform="scale(1.2)" xlink:href="#E284-MJMAINB-2D" x="2538" y="0"></use><use transform="scale(1.2)" xlink:href="#E284-MJMAINB-38" x="2921" y="0"></use><g transform="translate(4945,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">带</text></g><g transform="translate(5998,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">经</text></g><g transform="translate(7050,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">验</text></g><g transform="translate(8103,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">回</text></g><g transform="translate(9120,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">放</text></g><g transform="translate(10172,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">的</text></g><use transform="scale(1.2)" xlink:href="#E284-MJMAINB-51" x="9532" y="0"></use><g transform="translate(12725,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">学</text></g><g transform="translate(13778,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">习</text></g><g transform="translate(14795,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">最</text></g><g transform="translate(15847,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">优</text></g><g transform="translate(16900,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">策</text></g><g transform="translate(17953,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">略</text></g><g transform="translate(19006,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">求</text></g><g transform="translate(20059,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">解</text></g></g><g transform="translate(0,-11623)"><g transform="translate(-19,0)"><g transform="translate(0,7694)"><g><rect fill="black" stroke="none" width="1569" height="100" x="0" y="500"></rect></g></g><g transform="translate(0,-7495)"><g><rect fill="black" stroke="none" width="1569" height="100" x="0" y="-500"></rect></g></g></g><g transform="translate(1551,0)"><g transform="translate(0,7694)"><g><rect fill="black" stroke="none" width="41597" height="100" x="0" y="500"></rect></g></g><g transform="translate(0,6394)"><use xlink:href="#E284-MJMAIN-31"></use><use xlink:href="#E284-MJMAIN-2E" x="500" y="0"></use><g transform="translate(778,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(1608,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">初</text></g><g transform="translate(2439,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">始</text></g><g transform="translate(3269,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">化</text></g><g transform="translate(4100,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(4931,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">任</text></g><g transform="translate(5761,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">意</text></g><g transform="translate(6592,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">初</text></g><g transform="translate(7423,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">始</text></g><g transform="translate(8253,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">化</text></g><g transform="translate(9084,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">参</text></g><g transform="translate(9915,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">数</text></g><use xlink:href="#E284-MJMAINB-77" x="10995" y="0"></use><g transform="translate(11826,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">。</text></g></g></g><g transform="translate(0,5094)"><use xlink:href="#E284-MJMAIN-32"></use><use xlink:href="#E284-MJMAIN-2E" x="500" y="0"></use><g transform="translate(1472,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">对</text><g transform="translate(830,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">于</text></g><g transform="translate(1661,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">每</text></g><g transform="translate(2491,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">个</text></g><g transform="translate(3322,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">回</text></g><g transform="translate(4153,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">合</text></g><g transform="translate(4983,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">执</text></g><g transform="translate(5814,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">行</text></g><g transform="translate(6645,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">以</text></g><g transform="translate(7475,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">下</text></g><g transform="translate(8306,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">操</text></g><g transform="translate(9137,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">作</text></g><g transform="translate(9967,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">：</text></g></g></g><g transform="translate(0,3794)"><g transform="translate(2000,0)"><use xlink:href="#E284-MJMAIN-32"></use><use xlink:href="#E284-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E284-MJMAIN-31" x="778" y="0"></use><g transform="translate(1278,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(2108,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">初</text></g><g transform="translate(2939,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">始</text></g><g transform="translate(3769,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">化</text></g><g transform="translate(4600,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">状</text></g><g transform="translate(5431,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">态</text></g><g transform="translate(6261,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(7092,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">选</text></g><g transform="translate(7923,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">择</text></g><g transform="translate(8753,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">状</text></g><g transform="translate(9584,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">态</text></g><use xlink:href="#E284-MJMATHI-53" x="10665" y="0"></use><g transform="translate(11310,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">。</text></g></g></g></g><g transform="translate(0,2494)"><g transform="translate(2000,0)"><use xlink:href="#E284-MJMAIN-32"></use><use xlink:href="#E284-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E284-MJMAIN-32" x="778" y="0"></use><g transform="translate(1972,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">若</text><g transform="translate(830,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">回</text></g><g transform="translate(1661,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">合</text></g><g transform="translate(2491,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">未</text></g><g transform="translate(3322,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">结</text></g><g transform="translate(4153,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">束</text></g><g transform="translate(4983,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">，</text></g><g transform="translate(5814,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">执</text></g><g transform="translate(6645,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">行</text></g><g transform="translate(7475,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">以</text></g><g transform="translate(8306,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">下</text></g><g transform="translate(9137,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">操</text></g><g transform="translate(9967,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">作</text></g><g transform="translate(10798,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">：</text></g></g></g></g><g transform="translate(0,1185)"><g transform="translate(4000,0)"><use xlink:href="#E284-MJMAIN-32"></use><use xlink:href="#E284-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E284-MJMAIN-32" x="778" y="0"></use><use xlink:href="#E284-MJMAIN-2E" x="1278" y="0"></use><use xlink:href="#E284-MJMAIN-31" x="1556" y="0"></use><g transform="translate(2056,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(2886,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">采</text></g><g transform="translate(3717,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">样</text></g><g transform="translate(4547,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(5378,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">根</text></g><g transform="translate(6209,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">据</text></g><g transform="translate(7289,0)"><use xlink:href="#E284-MJMATHI-71" x="0" y="0"></use><use xlink:href="#E284-MJMAIN-28" x="460" y="0"></use><use xlink:href="#E284-MJMATHI-53" x="849" y="0"></use><use xlink:href="#E284-MJMAIN-2C" x="1494" y="0"></use><use xlink:href="#E284-MJMAIN-22C5" x="1938" y="0"></use><use xlink:href="#E284-MJMAIN-3B" x="2216" y="0"></use><use xlink:href="#E284-MJMAINB-77" x="2661" y="0"></use><use xlink:href="#E284-MJMAIN-29" x="3492" y="0"></use></g><g transform="translate(11171,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">选</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">择</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">并</text></g><g transform="translate(2741,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">执</text></g><g transform="translate(3572,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">行</text></g><g transform="translate(4403,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">动</text></g><g transform="translate(5233,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">作</text></g></g><use xlink:href="#E284-MJMATHI-41" x="17485" y="0"></use><g transform="translate(18235,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">，</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">观</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">测</text></g><g transform="translate(2741,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">得</text></g><g transform="translate(3572,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">到</text></g><g transform="translate(4403,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">的</text></g><g transform="translate(5233,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">奖</text></g><g transform="translate(6064,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">励</text></g></g><use xlink:href="#E284-MJMATHI-52" x="25381" y="0"></use><g transform="translate(26140,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">和</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">新</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">状</text></g><g transform="translate(2741,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">态</text></g></g><g transform="translate(29962,0)"><use xlink:href="#E284-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E284-MJMAIN-2032" x="925" y="583"></use></g><g transform="translate(30911,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">；</text></g></g></g></g><g transform="translate(0,-174)"><g transform="translate(4000,0)"><use xlink:href="#E284-MJMAIN-32"></use><use xlink:href="#E284-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E284-MJMAIN-32" x="778" y="0"></use><use xlink:href="#E284-MJMAIN-2E" x="1278" y="0"></use><use xlink:href="#E284-MJMAIN-32" x="1556" y="0"></use><g transform="translate(2056,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(2886,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">存</text></g><g transform="translate(3717,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">储</text></g><g transform="translate(4547,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(5378,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">将</text></g><g transform="translate(6209,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">经</text></g><g transform="translate(7039,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">验</text></g><g transform="translate(8120,0)"><use xlink:href="#E284-MJMAIN-28" x="0" y="0"></use><use xlink:href="#E284-MJMATHI-53" x="389" y="0"></use><use xlink:href="#E284-MJMAIN-2C" x="1034" y="0"></use><use xlink:href="#E284-MJMATHI-41" x="1478" y="0"></use><use xlink:href="#E284-MJMAIN-2C" x="2228" y="0"></use><use xlink:href="#E284-MJMATHI-52" x="2673" y="0"></use><use xlink:href="#E284-MJMAIN-2C" x="3432" y="0"></use><g transform="translate(3877,0)"><use xlink:href="#E284-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E284-MJMAIN-2032" x="925" y="583"></use></g><use xlink:href="#E284-MJMAIN-29" x="4826" y="0"></use></g><g transform="translate(13335,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">存</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">入</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">经</text></g><g transform="translate(2741,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">验</text></g><g transform="translate(3572,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">库</text></g><g transform="translate(4403,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">中</text></g><g transform="translate(5233,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">；</text></g></g></g></g><g transform="translate(0,-1524)"><g transform="translate(4000,0)"><use xlink:href="#E284-MJMAIN-32"></use><use xlink:href="#E284-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E284-MJMAIN-32" x="778" y="0"></use><use xlink:href="#E284-MJMAIN-2E" x="1278" y="0"></use><use xlink:href="#E284-MJMAIN-33" x="1556" y="0"></use><g transform="translate(2056,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(2886,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">回</text></g><g transform="translate(3717,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">放</text></g><g transform="translate(4547,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(5378,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">从</text></g><g transform="translate(6209,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">经</text></g><g transform="translate(7039,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">验</text></g><g transform="translate(7870,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">库</text></g><g transform="translate(8701,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">中</text></g><g transform="translate(9531,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">选</text></g><g transform="translate(10362,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">取</text></g><g transform="translate(11193,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">经</text></g><g transform="translate(12023,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">验</text></g><g transform="translate(13104,0)"><use xlink:href="#E284-MJMAIN-28" x="0" y="0"></use><g transform="translate(389,0)"><use xlink:href="#E284-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E284-MJMATHI-69" x="866" y="-213"></use></g><use xlink:href="#E284-MJMAIN-2C" x="1345" y="0"></use><g transform="translate(1790,0)"><use xlink:href="#E284-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E284-MJMATHI-69" x="1060" y="-213"></use></g><use xlink:href="#E284-MJMAIN-2C" x="2884" y="0"></use><g transform="translate(3329,0)"><use xlink:href="#E284-MJMATHI-52" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E284-MJMATHI-69" x="1073" y="-213"></use></g><use xlink:href="#E284-MJMAIN-2C" x="4432" y="0"></use><g transform="translate(4876,0)"><use xlink:href="#E284-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E284-MJMAIN-2032" x="925" y="444"></use><use transform="scale(0.707)" xlink:href="#E284-MJMATHI-69" x="866" y="-429"></use></g><use xlink:href="#E284-MJMAIN-29" x="5833" y="0"></use></g><g transform="translate(19327,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">；</text></g></g></g></g><g transform="translate(0,-2934)"><g transform="translate(4000,0)"><use xlink:href="#E284-MJMAIN-32"></use><use xlink:href="#E284-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E284-MJMAIN-32" x="778" y="0"></use><use xlink:href="#E284-MJMAIN-2E" x="1278" y="0"></use><use xlink:href="#E284-MJMAIN-34" x="1556" y="0"></use><g transform="translate(2056,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(2886,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">计</text></g><g transform="translate(3717,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">算</text></g><g transform="translate(4547,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">回</text></g><g transform="translate(5378,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">报</text></g><g transform="translate(6209,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">的</text></g><g transform="translate(7039,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">估</text></g><g transform="translate(7870,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">计</text></g><g transform="translate(8701,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">值</text></g><g transform="translate(9531,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(10362,0)"><use xlink:href="#E284-MJMATHI-55" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E284-MJMATHI-69" x="965" y="-213"></use><use xlink:href="#E284-MJMAIN-2190" x="1304" y="0"></use><g transform="translate(2582,0)"><use xlink:href="#E284-MJMATHI-52" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E284-MJMATHI-69" x="1073" y="-213"></use></g><use xlink:href="#E284-MJMAIN-2B" x="3907" y="0"></use><use xlink:href="#E284-MJMATHI-3B3" x="4907" y="0"></use><g transform="translate(5617,0)"><use xlink:href="#E284-MJMAIN-6D"></use><use xlink:href="#E284-MJMAIN-61" x="833" y="0"></use><use xlink:href="#E284-MJMAIN-78" x="1333" y="0"></use><use transform="scale(0.707)" xlink:href="#E284-MJMATHI-61" x="1051" y="-865"></use></g><use xlink:href="#E284-MJMATHI-71" x="7811" y="0"></use><use xlink:href="#E284-MJMAIN-28" x="8271" y="0"></use><g transform="translate(8660,0)"><use xlink:href="#E284-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E284-MJMAIN-2032" x="925" y="444"></use><use transform="scale(0.707)" xlink:href="#E284-MJMATHI-69" x="866" y="-429"></use></g><use xlink:href="#E284-MJMAIN-2C" x="9617" y="0"></use><use xlink:href="#E284-MJMATHI-61" x="10062" y="0"></use><use xlink:href="#E284-MJMAIN-3B" x="10591" y="0"></use><use xlink:href="#E284-MJMAINB-77" x="11036" y="0"></use><use xlink:href="#E284-MJMAIN-29" x="11867" y="0"></use></g><g transform="translate(22618,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">；</text></g></g></g></g><g transform="translate(0,-4836)"><g transform="translate(4000,0)"><use xlink:href="#E284-MJMAIN-32"></use><use xlink:href="#E284-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E284-MJMAIN-32" x="778" y="0"></use><use xlink:href="#E284-MJMAIN-2E" x="1278" y="0"></use><use xlink:href="#E284-MJMAIN-35" x="1556" y="0"></use><g transform="translate(2750,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">更</text><g transform="translate(830,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">新</text></g></g><use xlink:href="#E284-MJMAINB-77" x="4661" y="0"></use><g transform="translate(5492,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">以</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">减</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">小</text></g></g><g transform="translate(8484,0)"><use xlink:href="#E284-MJMAIN-5B" x="0" y="0"></use><g transform="translate(278,0)"><use xlink:href="#E284-MJMATHI-55" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E284-MJMATHI-69" x="965" y="-213"></use></g><use xlink:href="#E284-MJMAIN-2212" x="1527" y="0"></use><use xlink:href="#E284-MJMATHI-71" x="2527" y="0"></use><use xlink:href="#E284-MJMAIN-28" x="2987" y="0"></use><g transform="translate(3376,0)"><use xlink:href="#E284-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E284-MJMATHI-69" x="866" y="-213"></use></g><use xlink:href="#E284-MJMAIN-2C" x="4333" y="0"></use><g transform="translate(4778,0)"><use xlink:href="#E284-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E284-MJMATHI-69" x="1060" y="-213"></use></g><use xlink:href="#E284-MJMAIN-3B" x="5871" y="0"></use><use xlink:href="#E284-MJMAINB-77" x="6316" y="0"></use><use xlink:href="#E284-MJMAIN-29" x="7147" y="0"></use><g transform="translate(7536,0)"><use xlink:href="#E284-MJMAIN-5D" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E284-MJMAIN-32" x="393" y="583"></use></g></g><g transform="translate(16752,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">，</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">如</text></g></g><g transform="translate(18914,0)"><use xlink:href="#E284-MJMAINB-77" x="0" y="0"></use><use xlink:href="#E284-MJMAIN-2190" x="1108" y="0"></use><use xlink:href="#E284-MJMAINB-77" x="2386" y="0"></use><use xlink:href="#E284-MJMAIN-2B" x="3439" y="0"></use><use xlink:href="#E284-MJMATHI-3B1" x="4440" y="0"></use><use xlink:href="#E284-MJMAIN-5B" x="5080" y="0"></use><g transform="translate(5358,0)"><use xlink:href="#E284-MJMATHI-55" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E284-MJMATHI-69" x="965" y="-213"></use></g><use xlink:href="#E284-MJMAIN-2212" x="6607" y="0"></use><use xlink:href="#E284-MJMATHI-71" x="7607" y="0"></use><use xlink:href="#E284-MJMAIN-28" x="8067" y="0"></use><g transform="translate(8456,0)"><use xlink:href="#E284-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E284-MJMATHI-69" x="866" y="-213"></use></g><use xlink:href="#E284-MJMAIN-2C" x="9413" y="0"></use><g transform="translate(9858,0)"><use xlink:href="#E284-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E284-MJMATHI-69" x="1060" y="-213"></use></g><use xlink:href="#E284-MJMAIN-3B" x="10951" y="0"></use><use xlink:href="#E284-MJMAINB-77" x="11396" y="0"></use><use xlink:href="#E284-MJMAIN-29" x="12227" y="0"></use><use xlink:href="#E284-MJMAIN-5D" x="12616" y="0"></use><use xlink:href="#E284-MJMAIN-2207" x="12894" y="0"></use><use xlink:href="#E284-MJMATHI-71" x="13727" y="0"></use><use xlink:href="#E284-MJMAIN-28" x="14187" y="0"></use><g transform="translate(14576,0)"><use xlink:href="#E284-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E284-MJMATHI-69" x="866" y="-213"></use></g><use xlink:href="#E284-MJMAIN-2C" x="15533" y="0"></use><g transform="translate(15978,0)"><use xlink:href="#E284-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E284-MJMATHI-69" x="1060" y="-213"></use></g><use xlink:href="#E284-MJMAIN-3B" x="17072" y="0"></use><use xlink:href="#E284-MJMAINB-77" x="17516" y="0"></use><use xlink:href="#E284-MJMAIN-29" x="18347" y="0"></use></g><g transform="translate(37651,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">；</text></g></g></g></g><g transform="translate(0,-6195)"><g transform="translate(4000,0)"><use xlink:href="#E284-MJMAIN-32"></use><use xlink:href="#E284-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E284-MJMAIN-32" x="778" y="0"></use><use xlink:href="#E284-MJMAIN-2E" x="1278" y="0"></use><use xlink:href="#E284-MJMAIN-36" x="1556" y="0"></use><g transform="translate(2306,0)"><use xlink:href="#E284-MJMATHI-53" x="444" y="0"></use><use xlink:href="#E284-MJMAIN-2190" x="1367" y="0"></use><g transform="translate(2645,0)"><use xlink:href="#E284-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E284-MJMAIN-2032" x="925" y="583"></use></g></g><g transform="translate(5900,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">。</text></g></g></g></g><g transform="translate(0,-7495)"><g><rect fill="black" stroke="none" width="41597" height="100" x="0" y="-500"></rect></g></g></g></g></g></svg></span></div><script type="math/tex; mode=display" id="MathJax-Element-257">\; \\ \; \\
\large \textbf{算法 6-8   带经验回放的 Q 学习最优策略求解} \\
\begin{split}
\rule[5pt]{10mm}{0.1em} &\rule[5pt]{265mm}{0.1em} \\
&\text{1.（初始化）任意初始化参数 $\bold w$ 。} \\
&\text{2. $\;\,$对于每个回合执行以下操作：} \\
&\qquad \text{2.1（初始化状态）选择状态 $S$ 。} \\
&\qquad \text{2.2 $\;\,$若回合未结束，执行以下操作：} \\
&\qquad \qquad \text{2.2.1（采样）根据 $q(S,\cdot;\bold w)$ 选择并执行动作 $A$ ，观测得到的奖励 $R$ 和新状态 $S'$ ；} \\
&\qquad \qquad \text{2.2.2（存储）将经验 $(S,A,R,S')$ 存入经验库中；} \\
&\qquad \qquad \text{2.2.3（回放）从经验库中选取经验 $(S_i,A_i,R_i,S_i')$ ；} \\
&\qquad \qquad \text{2.2.4（计算回报的估计值）$U_i \leftarrow R_i + \gamma \max_a\, q(S_i',a;\bold w)$ ；} \\
&\qquad \qquad \text{2.2.5 $\;\,$更新 $\bold w$ 以减小 $[U_i-q(S_i,A_i;\bold w)]^2$ ，如 $\bold w \leftarrow \bold w + \alpha[U_i - q(S_i,A_i;\bold w)]\nabla q(S_i,A_i;\bold w)$ ；} \\
&\qquad \qquad \text{2.2.6 $\;\, S \leftarrow S'$ 。}\\
\rule[-5pt]{10mm}{0.1em} &\rule[-5pt]{265mm}{0.1em}
\end{split}
\; \\ \; \\</script></div></div><p><span>经验回放的好处有以下两点：</span></p><ul><li><span>在训练 Q 网络时，可以消除数据的关联，使得数据更像是独立同分布的（独立同分布是很多有监督学习的证明条件）；这样可以减小参数更新的方差，加快收敛。</span></li><li><span>能够重复使用经验，对于数据获取困难的情况尤其有用。</span></li></ul><p><span>从存储的角度，经验回放可以分为以下两种：</span></p><ul><li><strong><span>集中式回放：</span></strong><span>智能体在一个环境中运行，把经验同意存储在经验池中。</span></li><li><strong><span>分布式回放：</span></strong><span>智能体的多份拷贝（worker）同时在多个环境中运行，并将经验统一存储于经验池中。</span></li></ul><p><span>从采样的角度，经验回放又能分为以下两种：</span></p><ul><li><strong><span>均匀回放：</span></strong><span>等概率的从经验集中取经验。</span></li><li><strong><span>优先回放</span></strong><span>（Prioritized Experience Replay, PER）：为经验池里的每个经验指定一个优先级，在选取时更倾向于选取优先级高的经验。</span></li></ul><p><span>T. Schaul 等于 2016 年发表文字《Prioritized experience replay》提出了优先回放，其基本思想如上介绍，一般做法是，如果某个经验 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="0.801ex" height="1.839ex" viewBox="0 -706.9 345 791.9" role="img" focusable="false" style="vertical-align: -0.197ex;"><defs><path stroke-width="0" id="E336-MJMATHI-69" d="M184 600Q184 624 203 642T247 661Q265 661 277 649T290 619Q290 596 270 577T226 557Q211 557 198 567T184 600ZM21 287Q21 295 30 318T54 369T98 420T158 442Q197 442 223 419T250 357Q250 340 236 301T196 196T154 83Q149 61 149 51Q149 26 166 26Q175 26 185 29T208 43T235 78T260 137Q263 149 265 151T282 153Q302 153 302 143Q302 135 293 112T268 61T223 11T161 -11Q129 -11 102 10T74 74Q74 91 79 106T122 220Q160 321 166 341T173 380Q173 404 156 404H154Q124 404 99 371T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E336-MJMATHI-69" x="0" y="0"></use></g></svg></span><script type="math/tex">i</script><span> 的优先级为 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="2.058ex" height="1.746ex" viewBox="-39 -500.4 886 751.6" role="img" focusable="false" style="vertical-align: -0.583ex; margin-left: -0.091ex;"><defs><path stroke-width="0" id="E324-MJMATHI-70" d="M23 287Q24 290 25 295T30 317T40 348T55 381T75 411T101 433T134 442Q209 442 230 378L240 387Q302 442 358 442Q423 442 460 395T497 281Q497 173 421 82T249 -10Q227 -10 210 -4Q199 1 187 11T168 28L161 36Q160 35 139 -51T118 -138Q118 -144 126 -145T163 -148H188Q194 -155 194 -157T191 -175Q188 -187 185 -190T172 -194Q170 -194 161 -194T127 -193T65 -192Q-5 -192 -24 -194H-32Q-39 -187 -39 -183Q-37 -156 -26 -148H-6Q28 -147 33 -136Q36 -130 94 103T155 350Q156 355 156 364Q156 405 131 405Q109 405 94 377T71 316T59 280Q57 278 43 278H29Q23 284 23 287ZM178 102Q200 26 252 26Q282 26 310 49T356 107Q374 141 392 215T411 325V331Q411 405 350 405Q339 405 328 402T306 393T286 380T269 365T254 350T243 336T235 326L232 322Q232 321 229 308T218 264T204 212Q178 106 178 102Z"></path><path stroke-width="0" id="E324-MJMATHI-69" d="M184 600Q184 624 203 642T247 661Q265 661 277 649T290 619Q290 596 270 577T226 557Q211 557 198 567T184 600ZM21 287Q21 295 30 318T54 369T98 420T158 442Q197 442 223 419T250 357Q250 340 236 301T196 196T154 83Q149 61 149 51Q149 26 166 26Q175 26 185 29T208 43T235 78T260 137Q263 149 265 151T282 153Q302 153 302 143Q302 135 293 112T268 61T223 11T161 -11Q129 -11 102 10T74 74Q74 91 79 106T122 220Q160 321 166 341T173 380Q173 404 156 404H154Q124 404 99 371T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E324-MJMATHI-70" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E324-MJMATHI-69" x="711" y="-213"></use></g></svg></span><script type="math/tex">p_i</script><span> ，那么选取该经验的概率为 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="7.02ex" height="5.122ex" viewBox="0 -1164.9 3022.5 2205.2" role="img" focusable="false" style="vertical-align: -2.416ex;"><defs><path stroke-width="0" id="E325-MJMATHI-70" d="M23 287Q24 290 25 295T30 317T40 348T55 381T75 411T101 433T134 442Q209 442 230 378L240 387Q302 442 358 442Q423 442 460 395T497 281Q497 173 421 82T249 -10Q227 -10 210 -4Q199 1 187 11T168 28L161 36Q160 35 139 -51T118 -138Q118 -144 126 -145T163 -148H188Q194 -155 194 -157T191 -175Q188 -187 185 -190T172 -194Q170 -194 161 -194T127 -193T65 -192Q-5 -192 -24 -194H-32Q-39 -187 -39 -183Q-37 -156 -26 -148H-6Q28 -147 33 -136Q36 -130 94 103T155 350Q156 355 156 364Q156 405 131 405Q109 405 94 377T71 316T59 280Q57 278 43 278H29Q23 284 23 287ZM178 102Q200 26 252 26Q282 26 310 49T356 107Q374 141 392 215T411 325V331Q411 405 350 405Q339 405 328 402T306 393T286 380T269 365T254 350T243 336T235 326L232 322Q232 321 229 308T218 264T204 212Q178 106 178 102Z"></path><path stroke-width="0" id="E325-MJMATHI-69" d="M184 600Q184 624 203 642T247 661Q265 661 277 649T290 619Q290 596 270 577T226 557Q211 557 198 567T184 600ZM21 287Q21 295 30 318T54 369T98 420T158 442Q197 442 223 419T250 357Q250 340 236 301T196 196T154 83Q149 61 149 51Q149 26 166 26Q175 26 185 29T208 43T235 78T260 137Q263 149 265 151T282 153Q302 153 302 143Q302 135 293 112T268 61T223 11T161 -11Q129 -11 102 10T74 74Q74 91 79 106T122 220Q160 321 166 341T173 380Q173 404 156 404H154Q124 404 99 371T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Z"></path><path stroke-width="0" id="E325-MJSZ1-2211" d="M61 748Q64 750 489 750H913L954 640Q965 609 976 579T993 533T999 516H979L959 517Q936 579 886 621T777 682Q724 700 655 705T436 710H319Q183 710 183 709Q186 706 348 484T511 259Q517 250 513 244L490 216Q466 188 420 134T330 27L149 -187Q149 -188 362 -188Q388 -188 436 -188T506 -189Q679 -189 778 -162T936 -43Q946 -27 959 6H999L913 -249L489 -250Q65 -250 62 -248Q56 -246 56 -239Q56 -234 118 -161Q186 -81 245 -11L428 206Q428 207 242 462L57 717L56 728Q56 744 61 748Z"></path><path stroke-width="0" id="E325-MJMATHI-6B" d="M121 647Q121 657 125 670T137 683Q138 683 209 688T282 694Q294 694 294 686Q294 679 244 477Q194 279 194 272Q213 282 223 291Q247 309 292 354T362 415Q402 442 438 442Q468 442 485 423T503 369Q503 344 496 327T477 302T456 291T438 288Q418 288 406 299T394 328Q394 353 410 369T442 390L458 393Q446 405 434 405H430Q398 402 367 380T294 316T228 255Q230 254 243 252T267 246T293 238T320 224T342 206T359 180T365 147Q365 130 360 106T354 66Q354 26 381 26Q429 26 459 145Q461 153 479 153H483Q499 153 499 144Q499 139 496 130Q455 -11 378 -11Q333 -11 305 15T277 90Q277 108 280 121T283 145Q283 167 269 183T234 206T200 217T182 220H180Q168 178 159 139T145 81T136 44T129 20T122 7T111 -2Q98 -11 83 -11Q66 -11 57 -1T48 16Q48 26 85 176T158 471L195 616Q196 629 188 632T149 637H144Q134 637 131 637T124 640T121 647Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><g transform="translate(120,0)"><rect stroke="none" width="2782" height="60" x="0" y="220"></rect><g transform="translate(967,676)"><use xlink:href="#E325-MJMATHI-70" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E325-MJMATHI-69" x="711" y="-213"></use></g><g transform="translate(60,-694)"><use xlink:href="#E325-MJSZ1-2211" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E325-MJMATHI-6B" x="1493" y="-404"></use><g transform="translate(1691,0)"><use xlink:href="#E325-MJMATHI-70" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E325-MJMATHI-6B" x="711" y="-213"></use></g></g></g></g></svg></span><script type="math/tex">\displaystyle \frac{p_i}{\sum_k p_k}</script><span> 。</span></p><p><span>经验值的选取方法也有许多种，最常见的有：</span></p><ul><li><strong><span>成比例优先</span></strong><span>（Proportional priority）：第 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="0.801ex" height="1.839ex" viewBox="0 -706.9 345 791.9" role="img" focusable="false" style="vertical-align: -0.197ex;"><defs><path stroke-width="0" id="E336-MJMATHI-69" d="M184 600Q184 624 203 642T247 661Q265 661 277 649T290 619Q290 596 270 577T226 557Q211 557 198 567T184 600ZM21 287Q21 295 30 318T54 369T98 420T158 442Q197 442 223 419T250 357Q250 340 236 301T196 196T154 83Q149 61 149 51Q149 26 166 26Q175 26 185 29T208 43T235 78T260 137Q263 149 265 151T282 153Q302 153 302 143Q302 135 293 112T268 61T223 11T161 -11Q129 -11 102 10T74 74Q74 91 79 106T122 220Q160 321 166 341T173 380Q173 404 156 404H154Q124 404 99 371T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E336-MJMATHI-69" x="0" y="0"></use></g></svg></span><script type="math/tex">i</script><span> 个经验优先级为 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="13.997ex" height="2.71ex" viewBox="-39 -832.7 6026.5 1166.9" role="img" focusable="false" style="vertical-align: -0.776ex; margin-left: -0.091ex;"><defs><path stroke-width="0" id="E327-MJMATHI-70" d="M23 287Q24 290 25 295T30 317T40 348T55 381T75 411T101 433T134 442Q209 442 230 378L240 387Q302 442 358 442Q423 442 460 395T497 281Q497 173 421 82T249 -10Q227 -10 210 -4Q199 1 187 11T168 28L161 36Q160 35 139 -51T118 -138Q118 -144 126 -145T163 -148H188Q194 -155 194 -157T191 -175Q188 -187 185 -190T172 -194Q170 -194 161 -194T127 -193T65 -192Q-5 -192 -24 -194H-32Q-39 -187 -39 -183Q-37 -156 -26 -148H-6Q28 -147 33 -136Q36 -130 94 103T155 350Q156 355 156 364Q156 405 131 405Q109 405 94 377T71 316T59 280Q57 278 43 278H29Q23 284 23 287ZM178 102Q200 26 252 26Q282 26 310 49T356 107Q374 141 392 215T411 325V331Q411 405 350 405Q339 405 328 402T306 393T286 380T269 365T254 350T243 336T235 326L232 322Q232 321 229 308T218 264T204 212Q178 106 178 102Z"></path><path stroke-width="0" id="E327-MJMATHI-69" d="M184 600Q184 624 203 642T247 661Q265 661 277 649T290 619Q290 596 270 577T226 557Q211 557 198 567T184 600ZM21 287Q21 295 30 318T54 369T98 420T158 442Q197 442 223 419T250 357Q250 340 236 301T196 196T154 83Q149 61 149 51Q149 26 166 26Q175 26 185 29T208 43T235 78T260 137Q263 149 265 151T282 153Q302 153 302 143Q302 135 293 112T268 61T223 11T161 -11Q129 -11 102 10T74 74Q74 91 79 106T122 220Q160 321 166 341T173 380Q173 404 156 404H154Q124 404 99 371T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Z"></path><path stroke-width="0" id="E327-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E327-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E327-MJMATHI-3B4" d="M195 609Q195 656 227 686T302 717Q319 716 351 709T407 697T433 690Q451 682 451 662Q451 644 438 628T403 612Q382 612 348 641T288 671T249 657T235 628Q235 584 334 463Q401 379 401 292Q401 169 340 80T205 -10H198Q127 -10 83 36T36 153Q36 286 151 382Q191 413 252 434Q252 435 245 449T230 481T214 521T201 566T195 609ZM112 130Q112 83 136 55T204 27Q233 27 256 51T291 111T309 178T316 232Q316 267 309 298T295 344T269 400L259 396Q215 381 183 342T137 256T118 179T112 130Z"></path><path stroke-width="0" id="E327-MJMAIN-2B" d="M56 237T56 250T70 270H369V420L370 570Q380 583 389 583Q402 583 409 568V270H707Q722 262 722 250T707 230H409V-68Q401 -82 391 -82H389H387Q375 -82 369 -68V230H70Q56 237 56 250Z"></path><path stroke-width="0" id="E327-MJMATHI-3B5" d="M190 -22Q124 -22 76 11T27 107Q27 174 97 232L107 239L99 248Q76 273 76 304Q76 364 144 408T290 452H302Q360 452 405 421Q428 405 428 392Q428 381 417 369T391 356Q382 356 371 365T338 383T283 392Q217 392 167 368T116 308Q116 289 133 272Q142 263 145 262T157 264Q188 278 238 278H243Q308 278 308 247Q308 206 223 206Q177 206 142 219L132 212Q68 169 68 112Q68 39 201 39Q253 39 286 49T328 72T345 94T362 105Q376 103 376 88Q376 79 365 62T334 26T275 -8T190 -22Z"></path><path stroke-width="0" id="E327-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E327-MJMATHI-3B1" d="M34 156Q34 270 120 356T309 442Q379 442 421 402T478 304Q484 275 485 237V208Q534 282 560 374Q564 388 566 390T582 393Q603 393 603 385Q603 376 594 346T558 261T497 161L486 147L487 123Q489 67 495 47T514 26Q528 28 540 37T557 60Q559 67 562 68T577 70Q597 70 597 62Q597 56 591 43Q579 19 556 5T512 -10H505Q438 -10 414 62L411 69L400 61Q390 53 370 41T325 18T267 -2T203 -11Q124 -11 79 39T34 156ZM208 26Q257 26 306 47T379 90L403 112Q401 255 396 290Q382 405 304 405Q235 405 183 332Q156 292 139 224T121 120Q121 71 146 49T208 26Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E327-MJMATHI-70" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E327-MJMATHI-69" x="711" y="-213"></use><use xlink:href="#E327-MJMAIN-3D" x="1124" y="0"></use><use xlink:href="#E327-MJMAIN-28" x="2180" y="0"></use><g transform="translate(2569,0)"><use xlink:href="#E327-MJMATHI-3B4" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E327-MJMATHI-69" x="627" y="-213"></use></g><use xlink:href="#E327-MJMAIN-2B" x="3579" y="0"></use><use xlink:href="#E327-MJMATHI-3B5" x="4579" y="0"></use><g transform="translate(5045,0)"><use xlink:href="#E327-MJMAIN-29" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E327-MJMATHI-3B1" x="550" y="513"></use></g></g></svg></span><script type="math/tex">p_i = (\delta_i + \varepsilon)^\alpha</script><span> ，其中 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="1.83ex" height="2.324ex" viewBox="0 -791.1 788 1000.8" role="img" focusable="false" style="vertical-align: -0.487ex;"><defs><path stroke-width="0" id="E328-MJMATHI-3B4" d="M195 609Q195 656 227 686T302 717Q319 716 351 709T407 697T433 690Q451 682 451 662Q451 644 438 628T403 612Q382 612 348 641T288 671T249 657T235 628Q235 584 334 463Q401 379 401 292Q401 169 340 80T205 -10H198Q127 -10 83 36T36 153Q36 286 151 382Q191 413 252 434Q252 435 245 449T230 481T214 521T201 566T195 609ZM112 130Q112 83 136 55T204 27Q233 27 256 51T291 111T309 178T316 232Q316 267 309 298T295 344T269 400L259 396Q215 381 183 342T137 256T118 179T112 130Z"></path><path stroke-width="0" id="E328-MJMATHI-69" d="M184 600Q184 624 203 642T247 661Q265 661 277 649T290 619Q290 596 270 577T226 557Q211 557 198 567T184 600ZM21 287Q21 295 30 318T54 369T98 420T158 442Q197 442 223 419T250 357Q250 340 236 301T196 196T154 83Q149 61 149 51Q149 26 166 26Q175 26 185 29T208 43T235 78T260 137Q263 149 265 151T282 153Q302 153 302 143Q302 135 293 112T268 61T223 11T161 -11Q129 -11 102 10T74 74Q74 91 79 106T122 220Q160 321 166 341T173 380Q173 404 156 404H154Q124 404 99 371T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E328-MJMATHI-3B4" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E328-MJMATHI-69" x="627" y="-213"></use></g></svg></span><script type="math/tex">\delta_i</script><span> 是时序差分误差 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="21.865ex" height="2.71ex" viewBox="0 -832.7 9414.1 1166.9" role="img" focusable="false" style="vertical-align: -0.776ex;"><defs><path stroke-width="0" id="E329-MJMATHI-3B4" d="M195 609Q195 656 227 686T302 717Q319 716 351 709T407 697T433 690Q451 682 451 662Q451 644 438 628T403 612Q382 612 348 641T288 671T249 657T235 628Q235 584 334 463Q401 379 401 292Q401 169 340 80T205 -10H198Q127 -10 83 36T36 153Q36 286 151 382Q191 413 252 434Q252 435 245 449T230 481T214 521T201 566T195 609ZM112 130Q112 83 136 55T204 27Q233 27 256 51T291 111T309 178T316 232Q316 267 309 298T295 344T269 400L259 396Q215 381 183 342T137 256T118 179T112 130Z"></path><path stroke-width="0" id="E329-MJMATHI-69" d="M184 600Q184 624 203 642T247 661Q265 661 277 649T290 619Q290 596 270 577T226 557Q211 557 198 567T184 600ZM21 287Q21 295 30 318T54 369T98 420T158 442Q197 442 223 419T250 357Q250 340 236 301T196 196T154 83Q149 61 149 51Q149 26 166 26Q175 26 185 29T208 43T235 78T260 137Q263 149 265 151T282 153Q302 153 302 143Q302 135 293 112T268 61T223 11T161 -11Q129 -11 102 10T74 74Q74 91 79 106T122 220Q160 321 166 341T173 380Q173 404 156 404H154Q124 404 99 371T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Z"></path><path stroke-width="0" id="E329-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E329-MJMATHI-55" d="M107 637Q73 637 71 641Q70 643 70 649Q70 673 81 682Q83 683 98 683Q139 681 234 681Q268 681 297 681T342 682T362 682Q378 682 378 672Q378 670 376 658Q371 641 366 638H364Q362 638 359 638T352 638T343 637T334 637Q295 636 284 634T266 623Q265 621 238 518T184 302T154 169Q152 155 152 140Q152 86 183 55T269 24Q336 24 403 69T501 205L552 406Q599 598 599 606Q599 633 535 637Q511 637 511 648Q511 650 513 660Q517 676 519 679T529 683Q532 683 561 682T645 680Q696 680 723 681T752 682Q767 682 767 672Q767 650 759 642Q756 637 737 637Q666 633 648 597Q646 592 598 404Q557 235 548 205Q515 105 433 42T263 -22Q171 -22 116 34T60 167V183Q60 201 115 421Q164 622 164 628Q164 635 107 637Z"></path><path stroke-width="0" id="E329-MJMATHI-74" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path><path stroke-width="0" id="E329-MJMAIN-2212" d="M84 237T84 250T98 270H679Q694 262 694 250T679 230H98Q84 237 84 250Z"></path><path stroke-width="0" id="E329-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E329-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E329-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E329-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E329-MJMATHI-41" d="M208 74Q208 50 254 46Q272 46 272 35Q272 34 270 22Q267 8 264 4T251 0Q249 0 239 0T205 1T141 2Q70 2 50 0H42Q35 7 35 11Q37 38 48 46H62Q132 49 164 96Q170 102 345 401T523 704Q530 716 547 716H555H572Q578 707 578 706L606 383Q634 60 636 57Q641 46 701 46Q726 46 726 36Q726 34 723 22Q720 7 718 4T704 0Q701 0 690 0T651 1T578 2Q484 2 455 0H443Q437 6 437 9T439 27Q443 40 445 43L449 46H469Q523 49 533 63L521 213H283L249 155Q208 86 208 74ZM516 260Q516 271 504 416T490 562L463 519Q447 492 400 412L310 260L413 259Q516 259 516 260Z"></path><path stroke-width="0" id="E329-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E329-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E329-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E329-MJMATHI-3B4" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E329-MJMATHI-69" x="627" y="-213"></use><use xlink:href="#E329-MJMAIN-3D" x="1065" y="0"></use><g transform="translate(2121,0)"><use xlink:href="#E329-MJMATHI-55" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E329-MJMATHI-74" x="965" y="-213"></use></g><use xlink:href="#E329-MJMAIN-2212" x="3381" y="0"></use><use xlink:href="#E329-MJMATHI-71" x="4382" y="0"></use><use xlink:href="#E329-MJMAIN-28" x="4842" y="0"></use><g transform="translate(5231,0)"><use xlink:href="#E329-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E329-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E329-MJMAIN-2C" x="6199" y="0"></use><g transform="translate(6644,0)"><use xlink:href="#E329-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E329-MJMATHI-74" x="1060" y="-213"></use></g><use xlink:href="#E329-MJMAIN-3B" x="7749" y="0"></use><use xlink:href="#E329-MJMAINB-77" x="8194" y="0"></use><use xlink:href="#E329-MJMAIN-29" x="9025" y="0"></use></g></svg></span><script type="math/tex">\delta_i = U_t - q(S_t,A_t;\bold w)</script><span> 或 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="18.323ex" height="2.71ex" viewBox="0 -832.7 7889.1 1166.9" role="img" focusable="false" style="vertical-align: -0.776ex;"><defs><path stroke-width="0" id="E330-MJMATHI-3B4" d="M195 609Q195 656 227 686T302 717Q319 716 351 709T407 697T433 690Q451 682 451 662Q451 644 438 628T403 612Q382 612 348 641T288 671T249 657T235 628Q235 584 334 463Q401 379 401 292Q401 169 340 80T205 -10H198Q127 -10 83 36T36 153Q36 286 151 382Q191 413 252 434Q252 435 245 449T230 481T214 521T201 566T195 609ZM112 130Q112 83 136 55T204 27Q233 27 256 51T291 111T309 178T316 232Q316 267 309 298T295 344T269 400L259 396Q215 381 183 342T137 256T118 179T112 130Z"></path><path stroke-width="0" id="E330-MJMATHI-69" d="M184 600Q184 624 203 642T247 661Q265 661 277 649T290 619Q290 596 270 577T226 557Q211 557 198 567T184 600ZM21 287Q21 295 30 318T54 369T98 420T158 442Q197 442 223 419T250 357Q250 340 236 301T196 196T154 83Q149 61 149 51Q149 26 166 26Q175 26 185 29T208 43T235 78T260 137Q263 149 265 151T282 153Q302 153 302 143Q302 135 293 112T268 61T223 11T161 -11Q129 -11 102 10T74 74Q74 91 79 106T122 220Q160 321 166 341T173 380Q173 404 156 404H154Q124 404 99 371T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Z"></path><path stroke-width="0" id="E330-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E330-MJMATHI-55" d="M107 637Q73 637 71 641Q70 643 70 649Q70 673 81 682Q83 683 98 683Q139 681 234 681Q268 681 297 681T342 682T362 682Q378 682 378 672Q378 670 376 658Q371 641 366 638H364Q362 638 359 638T352 638T343 637T334 637Q295 636 284 634T266 623Q265 621 238 518T184 302T154 169Q152 155 152 140Q152 86 183 55T269 24Q336 24 403 69T501 205L552 406Q599 598 599 606Q599 633 535 637Q511 637 511 648Q511 650 513 660Q517 676 519 679T529 683Q532 683 561 682T645 680Q696 680 723 681T752 682Q767 682 767 672Q767 650 759 642Q756 637 737 637Q666 633 648 597Q646 592 598 404Q557 235 548 205Q515 105 433 42T263 -22Q171 -22 116 34T60 167V183Q60 201 115 421Q164 622 164 628Q164 635 107 637Z"></path><path stroke-width="0" id="E330-MJMATHI-74" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path><path stroke-width="0" id="E330-MJMAIN-2212" d="M84 237T84 250T98 270H679Q694 262 694 250T679 230H98Q84 237 84 250Z"></path><path stroke-width="0" id="E330-MJMATHI-76" d="M173 380Q173 405 154 405Q130 405 104 376T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Q21 294 29 316T53 368T97 419T160 441Q202 441 225 417T249 361Q249 344 246 335Q246 329 231 291T200 202T182 113Q182 86 187 69Q200 26 250 26Q287 26 319 60T369 139T398 222T409 277Q409 300 401 317T383 343T365 361T357 383Q357 405 376 424T417 443Q436 443 451 425T467 367Q467 340 455 284T418 159T347 40T241 -11Q177 -11 139 22Q102 54 102 117Q102 148 110 181T151 298Q173 362 173 380Z"></path><path stroke-width="0" id="E330-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E330-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E330-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E330-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E330-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E330-MJMATHI-3B4" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E330-MJMATHI-69" x="627" y="-213"></use><use xlink:href="#E330-MJMAIN-3D" x="1065" y="0"></use><g transform="translate(2121,0)"><use xlink:href="#E330-MJMATHI-55" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E330-MJMATHI-74" x="965" y="-213"></use></g><use xlink:href="#E330-MJMAIN-2212" x="3381" y="0"></use><use xlink:href="#E330-MJMATHI-76" x="4382" y="0"></use><use xlink:href="#E330-MJMAIN-28" x="4867" y="0"></use><g transform="translate(5256,0)"><use xlink:href="#E330-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E330-MJMATHI-74" x="866" y="-213"></use></g><use xlink:href="#E330-MJMAIN-3B" x="6224" y="0"></use><use xlink:href="#E330-MJMAINB-77" x="6669" y="0"></use><use xlink:href="#E330-MJMAIN-29" x="7500" y="0"></use></g></svg></span><script type="math/tex">\delta_i = U_t - v(S_t;\bold w)</script><span> ，</span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="1.082ex" height="1.36ex" viewBox="0 -500.4 466 585.5" role="img" focusable="false" style="vertical-align: -0.198ex;"><defs><path stroke-width="0" id="E331-MJMATHI-3B5" d="M190 -22Q124 -22 76 11T27 107Q27 174 97 232L107 239L99 248Q76 273 76 304Q76 364 144 408T290 452H302Q360 452 405 421Q428 405 428 392Q428 381 417 369T391 356Q382 356 371 365T338 383T283 392Q217 392 167 368T116 308Q116 289 133 272Q142 263 145 262T157 264Q188 278 238 278H243Q308 278 308 247Q308 206 223 206Q177 206 142 219L132 212Q68 169 68 112Q68 39 201 39Q253 39 286 49T328 72T345 94T362 105Q376 103 376 88Q376 79 365 62T334 26T275 -8T190 -22Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E331-MJMATHI-3B5" x="0" y="0"></use></g></svg></span><script type="math/tex">\varepsilon</script><span> 是预先选择的一个小正数，</span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="1.486ex" height="1.36ex" viewBox="0 -500.4 640 585.5" role="img" focusable="false" style="vertical-align: -0.198ex;"><defs><path stroke-width="0" id="E332-MJMATHI-3B1" d="M34 156Q34 270 120 356T309 442Q379 442 421 402T478 304Q484 275 485 237V208Q534 282 560 374Q564 388 566 390T582 393Q603 393 603 385Q603 376 594 346T558 261T497 161L486 147L487 123Q489 67 495 47T514 26Q528 28 540 37T557 60Q559 67 562 68T577 70Q597 70 597 62Q597 56 591 43Q579 19 556 5T512 -10H505Q438 -10 414 62L411 69L400 61Q390 53 370 41T325 18T267 -2T203 -11Q124 -11 79 39T34 156ZM208 26Q257 26 306 47T379 90L403 112Q401 255 396 290Q382 405 304 405Q235 405 183 332Q156 292 139 224T121 120Q121 71 146 49T208 26Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E332-MJMATHI-3B1" x="0" y="0"></use></g></svg></span><script type="math/tex">\alpha</script><span> 是正参数。</span></li><li><strong><span>基于排序优先</span></strong><span>（rank-based priority）：第 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="0.801ex" height="1.839ex" viewBox="0 -706.9 345 791.9" role="img" focusable="false" style="vertical-align: -0.197ex;"><defs><path stroke-width="0" id="E336-MJMATHI-69" d="M184 600Q184 624 203 642T247 661Q265 661 277 649T290 619Q290 596 270 577T226 557Q211 557 198 567T184 600ZM21 287Q21 295 30 318T54 369T98 420T158 442Q197 442 223 419T250 357Q250 340 236 301T196 196T154 83Q149 61 149 51Q149 26 166 26Q175 26 185 29T208 43T235 78T260 137Q263 149 265 151T282 153Q302 153 302 143Q302 135 293 112T268 61T223 11T161 -11Q129 -11 102 10T74 74Q74 91 79 106T122 220Q160 321 166 341T173 380Q173 404 156 404H154Q124 404 99 371T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E336-MJMATHI-69" x="0" y="0"></use></g></svg></span><script type="math/tex">i</script><span> 个经验的优先级为 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="16.372ex" height="5.894ex" viewBox="-39 -1538.7 7049 2537.5" role="img" focusable="false" style="vertical-align: -2.32ex; margin-left: -0.091ex;"><defs><path stroke-width="0" id="E334-MJMATHI-70" d="M23 287Q24 290 25 295T30 317T40 348T55 381T75 411T101 433T134 442Q209 442 230 378L240 387Q302 442 358 442Q423 442 460 395T497 281Q497 173 421 82T249 -10Q227 -10 210 -4Q199 1 187 11T168 28L161 36Q160 35 139 -51T118 -138Q118 -144 126 -145T163 -148H188Q194 -155 194 -157T191 -175Q188 -187 185 -190T172 -194Q170 -194 161 -194T127 -193T65 -192Q-5 -192 -24 -194H-32Q-39 -187 -39 -183Q-37 -156 -26 -148H-6Q28 -147 33 -136Q36 -130 94 103T155 350Q156 355 156 364Q156 405 131 405Q109 405 94 377T71 316T59 280Q57 278 43 278H29Q23 284 23 287ZM178 102Q200 26 252 26Q282 26 310 49T356 107Q374 141 392 215T411 325V331Q411 405 350 405Q339 405 328 402T306 393T286 380T269 365T254 350T243 336T235 326L232 322Q232 321 229 308T218 264T204 212Q178 106 178 102Z"></path><path stroke-width="0" id="E334-MJMATHI-69" d="M184 600Q184 624 203 642T247 661Q265 661 277 649T290 619Q290 596 270 577T226 557Q211 557 198 567T184 600ZM21 287Q21 295 30 318T54 369T98 420T158 442Q197 442 223 419T250 357Q250 340 236 301T196 196T154 83Q149 61 149 51Q149 26 166 26Q175 26 185 29T208 43T235 78T260 137Q263 149 265 151T282 153Q302 153 302 143Q302 135 293 112T268 61T223 11T161 -11Q129 -11 102 10T74 74Q74 91 79 106T122 220Q160 321 166 341T173 380Q173 404 156 404H154Q124 404 99 371T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Z"></path><path stroke-width="0" id="E334-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E334-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E334-MJMAIN-31" d="M213 578L200 573Q186 568 160 563T102 556H83V602H102Q149 604 189 617T245 641T273 663Q275 666 285 666Q294 666 302 660V361L303 61Q310 54 315 52T339 48T401 46H427V0H416Q395 3 257 3Q121 3 100 0H88V46H114Q136 46 152 46T177 47T193 50T201 52T207 57T213 61V578Z"></path><path stroke-width="0" id="E334-MJMATHI-72" d="M21 287Q22 290 23 295T28 317T38 348T53 381T73 411T99 433T132 442Q161 442 183 430T214 408T225 388Q227 382 228 382T236 389Q284 441 347 441H350Q398 441 422 400Q430 381 430 363Q430 333 417 315T391 292T366 288Q346 288 334 299T322 328Q322 376 378 392Q356 405 342 405Q286 405 239 331Q229 315 224 298T190 165Q156 25 151 16Q138 -11 108 -11Q95 -11 87 -5T76 7T74 17Q74 30 114 189T154 366Q154 405 128 405Q107 405 92 377T68 316T57 280Q55 278 41 278H27Q21 284 21 287Z"></path><path stroke-width="0" id="E334-MJMATHI-61" d="M33 157Q33 258 109 349T280 441Q331 441 370 392Q386 422 416 422Q429 422 439 414T449 394Q449 381 412 234T374 68Q374 43 381 35T402 26Q411 27 422 35Q443 55 463 131Q469 151 473 152Q475 153 483 153H487Q506 153 506 144Q506 138 501 117T481 63T449 13Q436 0 417 -8Q409 -10 393 -10Q359 -10 336 5T306 36L300 51Q299 52 296 50Q294 48 292 46Q233 -10 172 -10Q117 -10 75 30T33 157ZM351 328Q351 334 346 350T323 385T277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q217 26 254 59T298 110Q300 114 325 217T351 328Z"></path><path stroke-width="0" id="E334-MJMATHI-6E" d="M21 287Q22 293 24 303T36 341T56 388T89 425T135 442Q171 442 195 424T225 390T231 369Q231 367 232 367L243 378Q304 442 382 442Q436 442 469 415T503 336T465 179T427 52Q427 26 444 26Q450 26 453 27Q482 32 505 65T540 145Q542 153 560 153Q580 153 580 145Q580 144 576 130Q568 101 554 73T508 17T439 -10Q392 -10 371 17T350 73Q350 92 386 193T423 345Q423 404 379 404H374Q288 404 229 303L222 291L189 157Q156 26 151 16Q138 -11 108 -11Q95 -11 87 -5T76 7T74 17Q74 30 112 180T152 343Q153 348 153 366Q153 405 129 405Q91 405 66 305Q60 285 60 284Q58 278 41 278H27Q21 284 21 287Z"></path><path stroke-width="0" id="E334-MJMATHI-6B" d="M121 647Q121 657 125 670T137 683Q138 683 209 688T282 694Q294 694 294 686Q294 679 244 477Q194 279 194 272Q213 282 223 291Q247 309 292 354T362 415Q402 442 438 442Q468 442 485 423T503 369Q503 344 496 327T477 302T456 291T438 288Q418 288 406 299T394 328Q394 353 410 369T442 390L458 393Q446 405 434 405H430Q398 402 367 380T294 316T228 255Q230 254 243 252T267 246T293 238T320 224T342 206T359 180T365 147Q365 130 360 106T354 66Q354 26 381 26Q429 26 459 145Q461 153 479 153H483Q499 153 499 144Q499 139 496 130Q455 -11 378 -11Q333 -11 305 15T277 90Q277 108 280 121T283 145Q283 167 269 183T234 206T200 217T182 220H180Q168 178 159 139T145 81T136 44T129 20T122 7T111 -2Q98 -11 83 -11Q66 -11 57 -1T48 16Q48 26 85 176T158 471L195 616Q196 629 188 632T149 637H144Q134 637 131 637T124 640T121 647Z"></path><path stroke-width="0" id="E334-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E334-MJSZ3-28" d="M701 -940Q701 -943 695 -949H664Q662 -947 636 -922T591 -879T537 -818T475 -737T412 -636T350 -511T295 -362T250 -186T221 17T209 251Q209 962 573 1361Q596 1386 616 1405T649 1437T664 1450H695Q701 1444 701 1441Q701 1436 681 1415T629 1356T557 1261T476 1118T400 927T340 675T308 359Q306 321 306 250Q306 -139 400 -430T690 -924Q701 -936 701 -940Z"></path><path stroke-width="0" id="E334-MJSZ3-29" d="M34 1438Q34 1446 37 1448T50 1450H56H71Q73 1448 99 1423T144 1380T198 1319T260 1238T323 1137T385 1013T440 864T485 688T514 485T526 251Q526 134 519 53Q472 -519 162 -860Q139 -885 119 -904T86 -936T71 -949H56Q43 -949 39 -947T34 -937Q88 -883 140 -813Q428 -430 428 251Q428 453 402 628T338 922T245 1146T145 1309T46 1425Q44 1427 42 1429T39 1433T36 1436L34 1438Z"></path><path stroke-width="0" id="E334-MJMATHI-3B1" d="M34 156Q34 270 120 356T309 442Q379 442 421 402T478 304Q484 275 485 237V208Q534 282 560 374Q564 388 566 390T582 393Q603 393 603 385Q603 376 594 346T558 261T497 161L486 147L487 123Q489 67 495 47T514 26Q528 28 540 37T557 60Q559 67 562 68T577 70Q597 70 597 62Q597 56 591 43Q579 19 556 5T512 -10H505Q438 -10 414 62L411 69L400 61Q390 53 370 41T325 18T267 -2T203 -11Q124 -11 79 39T34 156ZM208 26Q257 26 306 47T379 90L403 112Q401 255 396 290Q382 405 304 405Q235 405 183 332Q156 292 139 224T121 120Q121 71 146 49T208 26Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E334-MJMATHI-70" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E334-MJMATHI-69" x="711" y="-213"></use><use xlink:href="#E334-MJMAIN-3D" x="1124" y="0"></use><g transform="translate(2180,0)"><use xlink:href="#E334-MJSZ3-28"></use><g transform="translate(736,0)"><g transform="translate(120,0)"><rect stroke="none" width="2564" height="60" x="0" y="220"></rect><use xlink:href="#E334-MJMAIN-31" x="1032" y="676"></use><g transform="translate(60,-686)"><use xlink:href="#E334-MJMATHI-72" x="0" y="0"></use><use xlink:href="#E334-MJMATHI-61" x="451" y="0"></use><use xlink:href="#E334-MJMATHI-6E" x="980" y="0"></use><g transform="translate(1580,0)"><use xlink:href="#E334-MJMATHI-6B" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E334-MJMATHI-69" x="736" y="-213"></use></g></g></g></g><use xlink:href="#E334-MJSZ3-29" x="3540" y="-1"></use><use transform="scale(0.707)" xlink:href="#E334-MJMATHI-3B1" x="6048" y="1663"></use></g></g></svg></span><script type="math/tex">\displaystyle p_i=\left(\frac{1}{rank_i}\right)^\alpha</script><span> 其中 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="5.679ex" height="2.224ex" viewBox="0 -748.4 2445 957.7" role="img" focusable="false" style="vertical-align: -0.486ex;"><defs><path stroke-width="0" id="E335-MJMATHI-72" d="M21 287Q22 290 23 295T28 317T38 348T53 381T73 411T99 433T132 442Q161 442 183 430T214 408T225 388Q227 382 228 382T236 389Q284 441 347 441H350Q398 441 422 400Q430 381 430 363Q430 333 417 315T391 292T366 288Q346 288 334 299T322 328Q322 376 378 392Q356 405 342 405Q286 405 239 331Q229 315 224 298T190 165Q156 25 151 16Q138 -11 108 -11Q95 -11 87 -5T76 7T74 17Q74 30 114 189T154 366Q154 405 128 405Q107 405 92 377T68 316T57 280Q55 278 41 278H27Q21 284 21 287Z"></path><path stroke-width="0" id="E335-MJMATHI-61" d="M33 157Q33 258 109 349T280 441Q331 441 370 392Q386 422 416 422Q429 422 439 414T449 394Q449 381 412 234T374 68Q374 43 381 35T402 26Q411 27 422 35Q443 55 463 131Q469 151 473 152Q475 153 483 153H487Q506 153 506 144Q506 138 501 117T481 63T449 13Q436 0 417 -8Q409 -10 393 -10Q359 -10 336 5T306 36L300 51Q299 52 296 50Q294 48 292 46Q233 -10 172 -10Q117 -10 75 30T33 157ZM351 328Q351 334 346 350T323 385T277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q217 26 254 59T298 110Q300 114 325 217T351 328Z"></path><path stroke-width="0" id="E335-MJMATHI-6E" d="M21 287Q22 293 24 303T36 341T56 388T89 425T135 442Q171 442 195 424T225 390T231 369Q231 367 232 367L243 378Q304 442 382 442Q436 442 469 415T503 336T465 179T427 52Q427 26 444 26Q450 26 453 27Q482 32 505 65T540 145Q542 153 560 153Q580 153 580 145Q580 144 576 130Q568 101 554 73T508 17T439 -10Q392 -10 371 17T350 73Q350 92 386 193T423 345Q423 404 379 404H374Q288 404 229 303L222 291L189 157Q156 26 151 16Q138 -11 108 -11Q95 -11 87 -5T76 7T74 17Q74 30 112 180T152 343Q153 348 153 366Q153 405 129 405Q91 405 66 305Q60 285 60 284Q58 278 41 278H27Q21 284 21 287Z"></path><path stroke-width="0" id="E335-MJMATHI-6B" d="M121 647Q121 657 125 670T137 683Q138 683 209 688T282 694Q294 694 294 686Q294 679 244 477Q194 279 194 272Q213 282 223 291Q247 309 292 354T362 415Q402 442 438 442Q468 442 485 423T503 369Q503 344 496 327T477 302T456 291T438 288Q418 288 406 299T394 328Q394 353 410 369T442 390L458 393Q446 405 434 405H430Q398 402 367 380T294 316T228 255Q230 254 243 252T267 246T293 238T320 224T342 206T359 180T365 147Q365 130 360 106T354 66Q354 26 381 26Q429 26 459 145Q461 153 479 153H483Q499 153 499 144Q499 139 496 130Q455 -11 378 -11Q333 -11 305 15T277 90Q277 108 280 121T283 145Q283 167 269 183T234 206T200 217T182 220H180Q168 178 159 139T145 81T136 44T129 20T122 7T111 -2Q98 -11 83 -11Q66 -11 57 -1T48 16Q48 26 85 176T158 471L195 616Q196 629 188 632T149 637H144Q134 637 131 637T124 640T121 647Z"></path><path stroke-width="0" id="E335-MJMATHI-69" d="M184 600Q184 624 203 642T247 661Q265 661 277 649T290 619Q290 596 270 577T226 557Q211 557 198 567T184 600ZM21 287Q21 295 30 318T54 369T98 420T158 442Q197 442 223 419T250 357Q250 340 236 301T196 196T154 83Q149 61 149 51Q149 26 166 26Q175 26 185 29T208 43T235 78T260 137Q263 149 265 151T282 153Q302 153 302 143Q302 135 293 112T268 61T223 11T161 -11Q129 -11 102 10T74 74Q74 91 79 106T122 220Q160 321 166 341T173 380Q173 404 156 404H154Q124 404 99 371T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E335-MJMATHI-72" x="0" y="0"></use><use xlink:href="#E335-MJMATHI-61" x="451" y="0"></use><use xlink:href="#E335-MJMATHI-6E" x="980" y="0"></use><g transform="translate(1580,0)"><use xlink:href="#E335-MJMATHI-6B" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E335-MJMATHI-69" x="736" y="-213"></use></g></g></svg></span><script type="math/tex">rank_i</script><span> 是第 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="0.801ex" height="1.839ex" viewBox="0 -706.9 345 791.9" role="img" focusable="false" style="vertical-align: -0.197ex;"><defs><path stroke-width="0" id="E336-MJMATHI-69" d="M184 600Q184 624 203 642T247 661Q265 661 277 649T290 619Q290 596 270 577T226 557Q211 557 198 567T184 600ZM21 287Q21 295 30 318T54 369T98 420T158 442Q197 442 223 419T250 357Q250 340 236 301T196 196T154 83Q149 61 149 51Q149 26 166 26Q175 26 185 29T208 43T235 78T260 137Q263 149 265 151T282 153Q302 153 302 143Q302 135 293 112T268 61T223 11T161 -11Q129 -11 102 10T74 74Q74 91 79 106T122 220Q160 321 166 341T173 380Q173 404 156 404H154Q124 404 99 371T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E336-MJMATHI-69" x="0" y="0"></use></g></svg></span><script type="math/tex">i</script><span> 个经验从大到小排序的排名，排名从 1 开始。</span></li></ul><p><span>D. Horgan 等在 2018 发表文章《Distributed prioritized experience replay》，将分布式经验回放和优先经验回放相结合，得到了</span><strong><span>分布式优先经验回放</span></strong><span>（distributed prioritized experience replay）。另外，由于经验回放会导致回合更新和多步学习算法无法使用，所以一般情况下是将经验回放用于 Q 学习。</span></p><p><span>对于基于自益的 Q 学习，其回报估计和动作价值的估计都和权重 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="1.93ex" height="1.36ex" viewBox="0 -500.4 831 585.5" role="img" focusable="false" style="vertical-align: -0.198ex;"><defs><path stroke-width="0" id="E337-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E337-MJMAINB-77" x="0" y="0"></use></g></svg></span><script type="math/tex">\bold w</script><span> 有关，当权重值变化时它们也会随着改变，而在学习的过程中，动作价值试图追逐一个变化的回报，也容易出现不稳定的情况。半梯度下降算法阻止对 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="2.411ex" height="2.228ex" viewBox="0 -749.6 1038.3 959.2" role="img" focusable="false" style="vertical-align: -0.487ex;"><defs><path stroke-width="0" id="E340-MJMATHI-55" d="M107 637Q73 637 71 641Q70 643 70 649Q70 673 81 682Q83 683 98 683Q139 681 234 681Q268 681 297 681T342 682T362 682Q378 682 378 672Q378 670 376 658Q371 641 366 638H364Q362 638 359 638T352 638T343 637T334 637Q295 636 284 634T266 623Q265 621 238 518T184 302T154 169Q152 155 152 140Q152 86 183 55T269 24Q336 24 403 69T501 205L552 406Q599 598 599 606Q599 633 535 637Q511 637 511 648Q511 650 513 660Q517 676 519 679T529 683Q532 683 561 682T645 680Q696 680 723 681T752 682Q767 682 767 672Q767 650 759 642Q756 637 737 637Q666 633 648 597Q646 592 598 404Q557 235 548 205Q515 105 433 42T263 -22Q171 -22 116 34T60 167V183Q60 201 115 421Q164 622 164 628Q164 635 107 637Z"></path><path stroke-width="0" id="E340-MJMATHI-74" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E340-MJMATHI-55" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E340-MJMATHI-74" x="965" y="-213"></use></g></svg></span><script type="math/tex">U_t</script><span> 求梯度能够解决该问题，其中一种阻止方法是将价值参数复制一份 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="4.891ex" height="1.842ex" viewBox="0 -500.4 2105.7 793.1" role="img" focusable="false" style="vertical-align: -0.68ex;"><defs><path stroke-width="0" id="E341-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E341-MJMAINB-77" x="0" y="0"></use><g transform="translate(831,-150)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(29.368) matrix(1 0 0 -1 0 0)">目</text><g transform="translate(587,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(29.368) matrix(1 0 0 -1 0 0)">标</text></g></g></g></svg></span><script type="math/tex">\bold w_{目标}</script><span> ，在计算 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="2.411ex" height="2.228ex" viewBox="0 -749.6 1038.3 959.2" role="img" focusable="false" style="vertical-align: -0.487ex;"><defs><path stroke-width="0" id="E340-MJMATHI-55" d="M107 637Q73 637 71 641Q70 643 70 649Q70 673 81 682Q83 683 98 683Q139 681 234 681Q268 681 297 681T342 682T362 682Q378 682 378 672Q378 670 376 658Q371 641 366 638H364Q362 638 359 638T352 638T343 637T334 637Q295 636 284 634T266 623Q265 621 238 518T184 302T154 169Q152 155 152 140Q152 86 183 55T269 24Q336 24 403 69T501 205L552 406Q599 598 599 606Q599 633 535 637Q511 637 511 648Q511 650 513 660Q517 676 519 679T529 683Q532 683 561 682T645 680Q696 680 723 681T752 682Q767 682 767 672Q767 650 759 642Q756 637 737 637Q666 633 648 597Q646 592 598 404Q557 235 548 205Q515 105 433 42T263 -22Q171 -22 116 34T60 167V183Q60 201 115 421Q164 622 164 628Q164 635 107 637Z"></path><path stroke-width="0" id="E340-MJMATHI-74" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E340-MJMATHI-55" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E340-MJMATHI-74" x="965" y="-213"></use></g></svg></span><script type="math/tex">U_t</script><span> 时用 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="4.891ex" height="1.842ex" viewBox="0 -500.4 2105.7 793.1" role="img" focusable="false" style="vertical-align: -0.68ex;"><defs><path stroke-width="0" id="E341-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E341-MJMAINB-77" x="0" y="0"></use><g transform="translate(831,-150)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(29.368) matrix(1 0 0 -1 0 0)">目</text><g transform="translate(587,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(29.368) matrix(1 0 0 -1 0 0)">标</text></g></g></g></svg></span><script type="math/tex">\bold w_{目标}</script><span> 计算；基于这一方法，V. Mnih 等在 2015 年发表论文《Human-level control through deep reinforcement learning》提出了</span><strong><span>目标网络</span></strong><span>（target network）这一概念，它是在原有的神经网络之外再搭建一份结构完全相同的网络，并将原先的网络称为</span><strong><span>评估网络</span></strong><span>（evaluation network）。</span></p><p><span>在学习过程中，目标网络用于自益求得回报估计作为学习目标；在权重更新过程中，先只更新评估网络的权重，在完成一定次数的更新后，再将评估网络的权重赋值给目标网络。这样由于使用目标网络得到的回报估计在一段时间内是相对稳定的，因此增加了学习的稳定性；目前目标网络也已经成为深度 Q 学习的主流做法。算法 6-9 为带目标网络的深度 Q 学习算法：</span></p><div contenteditable="false" spellcheck="false" class="mathjax-block md-end-block md-math-block md-rawblock" id="mathjax-n96" cid="n96" mdtype="math_block"><div class="md-rawblock-container md-math-container" tabindex="-1"><div class="MathJax_SVG_Display"><span class="MathJax_SVG" id="MathJax-Element-258-Frame" tabindex="-1" style="font-size: 100%; display: inline-block; zoom: 0.971345;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="101.294ex" height="53.739ex" viewBox="-18.1 -43.5 43612.5 23137.6" role="img" focusable="false" style="vertical-align: -53.638ex; margin-left: -0.042ex; max-width: 100%;"><defs><path stroke-width="0" id="E285-MJMAINB-36" d="M48 318Q48 395 68 456T120 553T193 613T273 646T350 655Q425 655 461 616T497 524Q497 485 475 468T428 451Q399 451 378 470T357 521Q357 565 403 588Q375 601 351 601Q313 601 282 584Q242 565 222 526Q199 473 199 367Q201 369 210 380T227 396T246 410T275 422T312 426Q438 426 494 332Q526 285 526 208V199Q526 112 465 53Q428 17 388 3T285 -11Q236 -11 195 7T135 43T104 80Q48 165 48 318ZM375 231V244V268Q375 295 373 310T364 342T341 366T299 374H297Q231 374 208 287Q200 257 200 196Q201 120 209 100Q231 47 288 47Q351 47 368 90Q375 112 375 231Z"></path><path stroke-width="0" id="E285-MJMAINB-2D" d="M13 166V278H318V166H13Z"></path><path stroke-width="0" id="E285-MJMAINB-39" d="M178 59Q206 48 238 48Q311 48 345 102Q370 138 375 259V278Q374 278 369 271T350 252T322 232Q297 220 258 220Q172 220 110 275T48 438V446Q54 561 146 618Q199 654 278 654Q321 654 329 653Q526 621 526 330Q526 252 507 190T457 92T388 31T312 -2T240 -11Q165 -11 121 25T77 120Q77 159 99 176T147 193T194 177T217 122Q217 113 216 106T211 92T205 82T198 73T191 67T184 62T178 59ZM374 446V465Q374 523 364 552T315 598Q309 600 293 601Q227 601 210 562Q199 539 199 433Q199 343 204 319T235 279Q250 272 274 271H282Q293 271 303 274T327 288T353 323T371 385Q374 403 374 446Z"></path><path stroke-width="0" id="E285-MJMAINB-51" d="M64 339Q64 431 96 502T182 614T295 675T420 696Q469 696 481 695Q620 680 709 589T798 339Q798 255 768 184Q720 77 611 26L600 21Q635 -26 682 -26H696Q769 -26 769 0Q769 7 774 12T787 18Q805 18 805 -7V-13Q803 -64 785 -106T737 -171Q720 -183 697 -191Q687 -193 668 -193Q636 -193 613 -182T575 -144T552 -94T532 -27Q531 -23 530 -16T528 -6T526 -3L512 -5Q499 -7 477 -8T431 -10Q393 -10 382 -9Q238 8 151 97T64 339ZM326 80Q326 113 356 138T430 163Q492 163 542 100L553 86Q554 85 561 91T578 108Q637 179 637 330Q637 430 619 498T548 604Q500 641 425 641Q408 641 390 637T347 623T299 590T259 535Q226 469 226 338Q226 244 246 180T318 79L325 74Q326 74 326 80ZM506 58Q480 112 433 112Q412 112 395 104T378 77Q378 44 431 44Q480 44 506 58Z"></path><path stroke-width="0" id="E285-MJMAIN-31" d="M213 578L200 573Q186 568 160 563T102 556H83V602H102Q149 604 189 617T245 641T273 663Q275 666 285 666Q294 666 302 660V361L303 61Q310 54 315 52T339 48T401 46H427V0H416Q395 3 257 3Q121 3 100 0H88V46H114Q136 46 152 46T177 47T193 50T201 52T207 57T213 61V578Z"></path><path stroke-width="0" id="E285-MJMAIN-2E" d="M78 60Q78 84 95 102T138 120Q162 120 180 104T199 61Q199 36 182 18T139 0T96 17T78 60Z"></path><path stroke-width="0" id="E285-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E285-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E285-MJMAIN-22C5" d="M78 250Q78 274 95 292T138 310Q162 310 180 294T199 251Q199 226 182 208T139 190T96 207T78 250Z"></path><path stroke-width="0" id="E285-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E285-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E285-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E285-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E285-MJMAIN-2190" d="M944 261T944 250T929 230H165Q167 228 182 216T211 189T244 152T277 96T303 25Q308 7 308 0Q308 -11 288 -11Q281 -11 278 -11T272 -7T267 2T263 21Q245 94 195 151T73 236Q58 242 55 247Q55 254 59 257T73 264Q121 283 158 314T215 375T247 434T264 480L267 497Q269 503 270 505T275 509T288 511Q308 511 308 500Q308 493 303 475Q293 438 278 406T246 352T215 315T185 287T165 270H929Q944 261 944 250Z"></path><path stroke-width="0" id="E285-MJMAIN-22EF" d="M78 250Q78 274 95 292T138 310Q162 310 180 294T199 251Q199 226 182 208T139 190T96 207T78 250ZM525 250Q525 274 542 292T585 310Q609 310 627 294T646 251Q646 226 629 208T586 190T543 207T525 250ZM972 250Q972 274 989 292T1032 310Q1056 310 1074 294T1093 251Q1093 226 1076 208T1033 190T990 207T972 250Z"></path><path stroke-width="0" id="E285-MJMAIN-36" d="M42 313Q42 476 123 571T303 666Q372 666 402 630T432 550Q432 525 418 510T379 495Q356 495 341 509T326 548Q326 592 373 601Q351 623 311 626Q240 626 194 566Q147 500 147 364L148 360Q153 366 156 373Q197 433 263 433H267Q313 433 348 414Q372 400 396 374T435 317Q456 268 456 210V192Q456 169 451 149Q440 90 387 34T253 -22Q225 -22 199 -14T143 16T92 75T56 172T42 313ZM257 397Q227 397 205 380T171 335T154 278T148 216Q148 133 160 97T198 39Q222 21 251 21Q302 21 329 59Q342 77 347 104T352 209Q352 289 347 316T329 361Q302 397 257 397Z"></path><path stroke-width="0" id="E285-MJMAIN-2D" d="M11 179V252H277V179H11Z"></path><path stroke-width="0" id="E285-MJMAIN-38" d="M70 417T70 494T124 618T248 666Q319 666 374 624T429 515Q429 485 418 459T392 417T361 389T335 371T324 363L338 354Q352 344 366 334T382 323Q457 264 457 174Q457 95 399 37T249 -22Q159 -22 101 29T43 155Q43 263 172 335L154 348Q133 361 127 368Q70 417 70 494ZM286 386L292 390Q298 394 301 396T311 403T323 413T334 425T345 438T355 454T364 471T369 491T371 513Q371 556 342 586T275 624Q268 625 242 625Q201 625 165 599T128 534Q128 511 141 492T167 463T217 431Q224 426 228 424L286 386ZM250 21Q308 21 350 55T392 137Q392 154 387 169T375 194T353 216T330 234T301 253T274 270Q260 279 244 289T218 306L210 311Q204 311 181 294T133 239T107 157Q107 98 150 60T250 21Z"></path><path stroke-width="0" id="E285-MJMAIN-32" d="M109 429Q82 429 66 447T50 491Q50 562 103 614T235 666Q326 666 387 610T449 465Q449 422 429 383T381 315T301 241Q265 210 201 149L142 93L218 92Q375 92 385 97Q392 99 409 186V189H449V186Q448 183 436 95T421 3V0H50V19V31Q50 38 56 46T86 81Q115 113 136 137Q145 147 170 174T204 211T233 244T261 278T284 308T305 340T320 369T333 401T340 431T343 464Q343 527 309 573T212 619Q179 619 154 602T119 569T109 550Q109 549 114 549Q132 549 151 535T170 489Q170 464 154 447T109 429Z"></path><path stroke-width="0" id="E285-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E285-MJMATHI-41" d="M208 74Q208 50 254 46Q272 46 272 35Q272 34 270 22Q267 8 264 4T251 0Q249 0 239 0T205 1T141 2Q70 2 50 0H42Q35 7 35 11Q37 38 48 46H62Q132 49 164 96Q170 102 345 401T523 704Q530 716 547 716H555H572Q578 707 578 706L606 383Q634 60 636 57Q641 46 701 46Q726 46 726 36Q726 34 723 22Q720 7 718 4T704 0Q701 0 690 0T651 1T578 2Q484 2 455 0H443Q437 6 437 9T439 27Q443 40 445 43L449 46H469Q523 49 533 63L521 213H283L249 155Q208 86 208 74ZM516 260Q516 271 504 416T490 562L463 519Q447 492 400 412L310 260L413 259Q516 259 516 260Z"></path><path stroke-width="0" id="E285-MJMATHI-52" d="M230 637Q203 637 198 638T193 649Q193 676 204 682Q206 683 378 683Q550 682 564 680Q620 672 658 652T712 606T733 563T739 529Q739 484 710 445T643 385T576 351T538 338L545 333Q612 295 612 223Q612 212 607 162T602 80V71Q602 53 603 43T614 25T640 16Q668 16 686 38T712 85Q717 99 720 102T735 105Q755 105 755 93Q755 75 731 36Q693 -21 641 -21H632Q571 -21 531 4T487 82Q487 109 502 166T517 239Q517 290 474 313Q459 320 449 321T378 323H309L277 193Q244 61 244 59Q244 55 245 54T252 50T269 48T302 46H333Q339 38 339 37T336 19Q332 6 326 0H311Q275 2 180 2Q146 2 117 2T71 2T50 1Q33 1 33 10Q33 12 36 24Q41 43 46 45Q50 46 61 46H67Q94 46 127 49Q141 52 146 61Q149 65 218 339T287 628Q287 635 230 637ZM630 554Q630 586 609 608T523 636Q521 636 500 636T462 637H440Q393 637 386 627Q385 624 352 494T319 361Q319 360 388 360Q466 361 492 367Q556 377 592 426Q608 449 619 486T630 554Z"></path><path stroke-width="0" id="E285-MJMAIN-2032" d="M79 43Q73 43 52 49T30 61Q30 68 85 293T146 528Q161 560 198 560Q218 560 240 545T262 501Q262 496 260 486Q259 479 173 263T84 45T79 43Z"></path><path stroke-width="0" id="E285-MJCAL-44" d="M37 475Q19 475 19 487Q19 536 103 604T327 682H356Q386 683 408 683H419Q475 683 506 681T582 668T667 633Q766 571 766 450Q766 365 723 287T611 152T455 57T279 6Q248 1 160 0Q148 0 131 0T108 -1Q72 -1 72 11Q72 24 90 40T133 64L144 68L152 88Q247 328 272 587Q275 613 272 613Q272 613 269 613Q225 610 195 602T149 579T129 556T119 532Q118 530 116 525T113 518Q102 502 80 490T37 475ZM665 407Q665 596 412 613Q403 614 383 614Q370 614 370 612Q370 598 363 542T323 357T242 103L228 69H265Q391 73 481 119Q536 148 575 188T633 268T658 338T665 392V407Z"></path><path stroke-width="0" id="E285-MJMAIN-33" d="M127 463Q100 463 85 480T69 524Q69 579 117 622T233 665Q268 665 277 664Q351 652 390 611T430 522Q430 470 396 421T302 350L299 348Q299 347 308 345T337 336T375 315Q457 262 457 175Q457 96 395 37T238 -22Q158 -22 100 21T42 130Q42 158 60 175T105 193Q133 193 151 175T169 130Q169 119 166 110T159 94T148 82T136 74T126 70T118 67L114 66Q165 21 238 21Q293 21 321 74Q338 107 338 175V195Q338 290 274 322Q259 328 213 329L171 330L168 332Q166 335 166 348Q166 366 174 366Q202 366 232 371Q266 376 294 413T322 525V533Q322 590 287 612Q265 626 240 626Q208 626 181 615T143 592T132 580H135Q138 579 143 578T153 573T165 566T175 555T183 540T186 520Q186 498 172 481T127 463Z"></path><path stroke-width="0" id="E285-MJMATHI-69" d="M184 600Q184 624 203 642T247 661Q265 661 277 649T290 619Q290 596 270 577T226 557Q211 557 198 567T184 600ZM21 287Q21 295 30 318T54 369T98 420T158 442Q197 442 223 419T250 357Q250 340 236 301T196 196T154 83Q149 61 149 51Q149 26 166 26Q175 26 185 29T208 43T235 78T260 137Q263 149 265 151T282 153Q302 153 302 143Q302 135 293 112T268 61T223 11T161 -11Q129 -11 102 10T74 74Q74 91 79 106T122 220Q160 321 166 341T173 380Q173 404 156 404H154Q124 404 99 371T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Z"></path><path stroke-width="0" id="E285-MJMAIN-2208" d="M84 250Q84 372 166 450T360 539Q361 539 377 539T419 540T469 540H568Q583 532 583 520Q583 511 570 501L466 500Q355 499 329 494Q280 482 242 458T183 409T147 354T129 306T124 272V270H568Q583 262 583 250T568 230H124V228Q124 207 134 177T167 112T231 48T328 7Q355 1 466 0H570Q583 -10 583 -20Q583 -32 568 -40H471Q464 -40 446 -40T417 -41Q262 -41 172 45Q84 127 84 250Z"></path><path stroke-width="0" id="E285-MJCAL-42" d="M304 342Q292 342 292 353Q292 372 323 391Q331 396 417 428T533 487Q563 512 563 555V562Q563 575 557 589T530 618T475 636Q429 636 396 613T330 539Q263 446 210 238Q196 183 173 120Q135 31 121 16Q108 1 85 -10T47 -22T32 -10Q32 -5 44 18T77 93T112 206Q135 296 154 395T182 550T191 615Q191 616 190 616Q188 616 179 611T157 601T131 594Q113 594 113 605Q113 623 144 644Q154 650 205 676T267 703Q277 705 279 705Q295 705 295 693Q295 686 288 635T278 575Q278 572 287 582Q336 635 402 669T540 704Q603 704 633 673T664 599Q664 559 638 523T580 462Q553 440 504 413L491 407L504 402Q566 381 596 338T627 244Q627 172 575 110T444 13T284 -22Q208 -22 158 28Q144 42 146 50Q150 67 178 85T230 103Q236 103 246 95T267 75T302 56T357 47Q436 47 486 93Q526 136 526 198V210Q526 228 518 249T491 292T436 330T350 345Q335 345 321 344T304 342Z"></path><path stroke-width="0" id="E285-MJMAIN-34" d="M462 0Q444 3 333 3Q217 3 199 0H190V46H221Q241 46 248 46T265 48T279 53T286 61Q287 63 287 115V165H28V211L179 442Q332 674 334 675Q336 677 355 677H373L379 671V211H471V165H379V114Q379 73 379 66T385 54Q393 47 442 46H471V0H462ZM293 211V545L74 212L183 211H293Z"></path><path stroke-width="0" id="E285-MJMATHI-55" d="M107 637Q73 637 71 641Q70 643 70 649Q70 673 81 682Q83 683 98 683Q139 681 234 681Q268 681 297 681T342 682T362 682Q378 682 378 672Q378 670 376 658Q371 641 366 638H364Q362 638 359 638T352 638T343 637T334 637Q295 636 284 634T266 623Q265 621 238 518T184 302T154 169Q152 155 152 140Q152 86 183 55T269 24Q336 24 403 69T501 205L552 406Q599 598 599 606Q599 633 535 637Q511 637 511 648Q511 650 513 660Q517 676 519 679T529 683Q532 683 561 682T645 680Q696 680 723 681T752 682Q767 682 767 672Q767 650 759 642Q756 637 737 637Q666 633 648 597Q646 592 598 404Q557 235 548 205Q515 105 433 42T263 -22Q171 -22 116 34T60 167V183Q60 201 115 421Q164 622 164 628Q164 635 107 637Z"></path><path stroke-width="0" id="E285-MJMAIN-2B" d="M56 237T56 250T70 270H369V420L370 570Q380 583 389 583Q402 583 409 568V270H707Q722 262 722 250T707 230H409V-68Q401 -82 391 -82H389H387Q375 -82 369 -68V230H70Q56 237 56 250Z"></path><path stroke-width="0" id="E285-MJMATHI-3B3" d="M31 249Q11 249 11 258Q11 275 26 304T66 365T129 418T206 441Q233 441 239 440Q287 429 318 386T371 255Q385 195 385 170Q385 166 386 166L398 193Q418 244 443 300T486 391T508 430Q510 431 524 431H537Q543 425 543 422Q543 418 522 378T463 251T391 71Q385 55 378 6T357 -100Q341 -165 330 -190T303 -216Q286 -216 286 -188Q286 -138 340 32L346 51L347 69Q348 79 348 100Q348 257 291 317Q251 355 196 355Q148 355 108 329T51 260Q49 251 47 251Q45 249 31 249Z"></path><path stroke-width="0" id="E285-MJMAIN-6D" d="M41 46H55Q94 46 102 60V68Q102 77 102 91T102 122T103 161T103 203Q103 234 103 269T102 328V351Q99 370 88 376T43 385H25V408Q25 431 27 431L37 432Q47 433 65 434T102 436Q119 437 138 438T167 441T178 442H181V402Q181 364 182 364T187 369T199 384T218 402T247 421T285 437Q305 442 336 442Q351 442 364 440T387 434T406 426T421 417T432 406T441 395T448 384T452 374T455 366L457 361L460 365Q463 369 466 373T475 384T488 397T503 410T523 422T546 432T572 439T603 442Q729 442 740 329Q741 322 741 190V104Q741 66 743 59T754 49Q775 46 803 46H819V0H811L788 1Q764 2 737 2T699 3Q596 3 587 0H579V46H595Q656 46 656 62Q657 64 657 200Q656 335 655 343Q649 371 635 385T611 402T585 404Q540 404 506 370Q479 343 472 315T464 232V168V108Q464 78 465 68T468 55T477 49Q498 46 526 46H542V0H534L510 1Q487 2 460 2T422 3Q319 3 310 0H302V46H318Q379 46 379 62Q380 64 380 200Q379 335 378 343Q372 371 358 385T334 402T308 404Q263 404 229 370Q202 343 195 315T187 232V168V108Q187 78 188 68T191 55T200 49Q221 46 249 46H265V0H257L234 1Q210 2 183 2T145 3Q42 3 33 0H25V46H41Z"></path><path stroke-width="0" id="E285-MJMAIN-61" d="M137 305T115 305T78 320T63 359Q63 394 97 421T218 448Q291 448 336 416T396 340Q401 326 401 309T402 194V124Q402 76 407 58T428 40Q443 40 448 56T453 109V145H493V106Q492 66 490 59Q481 29 455 12T400 -6T353 12T329 54V58L327 55Q325 52 322 49T314 40T302 29T287 17T269 6T247 -2T221 -8T190 -11Q130 -11 82 20T34 107Q34 128 41 147T68 188T116 225T194 253T304 268H318V290Q318 324 312 340Q290 411 215 411Q197 411 181 410T156 406T148 403Q170 388 170 359Q170 334 154 320ZM126 106Q126 75 150 51T209 26Q247 26 276 49T315 109Q317 116 318 175Q318 233 317 233Q309 233 296 232T251 223T193 203T147 166T126 106Z"></path><path stroke-width="0" id="E285-MJMAIN-78" d="M201 0Q189 3 102 3Q26 3 17 0H11V46H25Q48 47 67 52T96 61T121 78T139 96T160 122T180 150L226 210L168 288Q159 301 149 315T133 336T122 351T113 363T107 370T100 376T94 379T88 381T80 383Q74 383 44 385H16V431H23Q59 429 126 429Q219 429 229 431H237V385Q201 381 201 369Q201 367 211 353T239 315T268 274L272 270L297 304Q329 345 329 358Q329 364 327 369T322 376T317 380T310 384L307 385H302V431H309Q324 428 408 428Q487 428 493 431H499V385H492Q443 385 411 368Q394 360 377 341T312 257L296 236L358 151Q424 61 429 57T446 50Q464 46 499 46H516V0H510H502Q494 1 482 1T457 2T432 2T414 3Q403 3 377 3T327 1L304 0H295V46H298Q309 46 320 51T331 63Q331 65 291 120L250 175Q249 174 219 133T185 88Q181 83 181 74Q181 63 188 55T206 46Q208 46 208 23V0H201Z"></path><path stroke-width="0" id="E285-MJMATHI-61" d="M33 157Q33 258 109 349T280 441Q331 441 370 392Q386 422 416 422Q429 422 439 414T449 394Q449 381 412 234T374 68Q374 43 381 35T402 26Q411 27 422 35Q443 55 463 131Q469 151 473 152Q475 153 483 153H487Q506 153 506 144Q506 138 501 117T481 63T449 13Q436 0 417 -8Q409 -10 393 -10Q359 -10 336 5T306 36L300 51Q299 52 296 50Q294 48 292 46Q233 -10 172 -10Q117 -10 75 30T33 157ZM351 328Q351 334 346 350T323 385T277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q217 26 254 59T298 110Q300 114 325 217T351 328Z"></path><path stroke-width="0" id="E285-MJMAIN-35" d="M164 157Q164 133 148 117T109 101H102Q148 22 224 22Q294 22 326 82Q345 115 345 210Q345 313 318 349Q292 382 260 382H254Q176 382 136 314Q132 307 129 306T114 304Q97 304 95 310Q93 314 93 485V614Q93 664 98 664Q100 666 102 666Q103 666 123 658T178 642T253 634Q324 634 389 662Q397 666 402 666Q410 666 410 648V635Q328 538 205 538Q174 538 149 544L139 546V374Q158 388 169 396T205 412T256 420Q337 420 393 355T449 201Q449 109 385 44T229 -22Q148 -22 99 32T50 154Q50 178 61 192T84 210T107 214Q132 214 148 197T164 157Z"></path><path stroke-width="0" id="E285-MJMAIN-7C" d="M139 -249H137Q125 -249 119 -235V251L120 737Q130 750 139 750Q152 750 159 735V-235Q151 -249 141 -249H139Z"></path><path stroke-width="0" id="E285-MJSZ2-2211" d="M60 948Q63 950 665 950H1267L1325 815Q1384 677 1388 669H1348L1341 683Q1320 724 1285 761Q1235 809 1174 838T1033 881T882 898T699 902H574H543H251L259 891Q722 258 724 252Q725 250 724 246Q721 243 460 -56L196 -356Q196 -357 407 -357Q459 -357 548 -357T676 -358Q812 -358 896 -353T1063 -332T1204 -283T1307 -196Q1328 -170 1348 -124H1388Q1388 -125 1381 -145T1356 -210T1325 -294L1267 -449L666 -450Q64 -450 61 -448Q55 -446 55 -439Q55 -437 57 -433L590 177Q590 178 557 222T452 366T322 544L56 909L55 924Q55 945 60 948Z"></path><path stroke-width="0" id="E285-MJMAIN-5B" d="M118 -250V750H255V710H158V-210H255V-250H118Z"></path><path stroke-width="0" id="E285-MJMAIN-2212" d="M84 237T84 250T98 270H679Q694 262 694 250T679 230H98Q84 237 84 250Z"></path><path stroke-width="0" id="E285-MJMAIN-5D" d="M22 710V750H159V-250H22V-210H119V710H22Z"></path><path stroke-width="0" id="E285-MJMATHI-3B1" d="M34 156Q34 270 120 356T309 442Q379 442 421 402T478 304Q484 275 485 237V208Q534 282 560 374Q564 388 566 390T582 393Q603 393 603 385Q603 376 594 346T558 261T497 161L486 147L487 123Q489 67 495 47T514 26Q528 28 540 37T557 60Q559 67 562 68T577 70Q597 70 597 62Q597 56 591 43Q579 19 556 5T512 -10H505Q438 -10 414 62L411 69L400 61Q390 53 370 41T325 18T267 -2T203 -11Q124 -11 79 39T34 156ZM208 26Q257 26 306 47T379 90L403 112Q401 255 396 290Q382 405 304 405Q235 405 183 332Q156 292 139 224T121 120Q121 71 146 49T208 26Z"></path><path stroke-width="0" id="E285-MJMAIN-2207" d="M46 676Q46 679 51 683H781Q786 679 786 676Q786 674 617 326T444 -26Q439 -33 416 -33T388 -26Q385 -22 216 326T46 676ZM697 596Q697 597 445 597T193 596Q195 591 319 336T445 80L697 596Z"></path><path stroke-width="0" id="E285-MJMAIN-37" d="M55 458Q56 460 72 567L88 674Q88 676 108 676H128V672Q128 662 143 655T195 646T364 644H485V605L417 512Q408 500 387 472T360 435T339 403T319 367T305 330T292 284T284 230T278 162T275 80Q275 66 275 52T274 28V19Q270 2 255 -10T221 -22Q210 -22 200 -19T179 0T168 40Q168 198 265 368Q285 400 349 489L395 552H302Q128 552 119 546Q113 543 108 522T98 479L95 458V455H55V458Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><g transform="translate(6974,-2466)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">算</text><g transform="translate(1052,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">法</text></g><use transform="scale(1.2)" xlink:href="#E285-MJMAINB-36" x="1963" y="0"></use><use transform="scale(1.2)" xlink:href="#E285-MJMAINB-2D" x="2538" y="0"></use><use transform="scale(1.2)" xlink:href="#E285-MJMAINB-39" x="2921" y="0"></use><g transform="translate(4945,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">带</text></g><g transform="translate(5998,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">经</text></g><g transform="translate(7050,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">验</text></g><g transform="translate(8103,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">回</text></g><g transform="translate(9120,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">放</text></g><g transform="translate(10172,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">和</text></g><g transform="translate(11189,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">目</text></g><g transform="translate(12205,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">标</text></g><g transform="translate(13258,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">网</text></g><g transform="translate(14274,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">络</text></g><g transform="translate(15327,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">的</text></g><g transform="translate(16343,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">深</text></g><g transform="translate(17396,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">度</text></g><use transform="scale(1.2)" xlink:href="#E285-MJMAINB-51" x="15582" y="0"></use><g transform="translate(19986,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">学</text></g><g transform="translate(21038,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">习</text></g><g transform="translate(22055,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">最</text></g><g transform="translate(23108,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">优</text></g><g transform="translate(24160,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">策</text></g><g transform="translate(25213,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">略</text></g><g transform="translate(26266,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">求</text></g><g transform="translate(27319,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" font-weight="bold" stroke="none" transform="scale(49.839) matrix(1 0 0 -1 0 0)">解</text></g></g><g transform="translate(0,-12512)"><g transform="translate(-19,0)"><g transform="translate(0,8584)"><g><rect fill="black" stroke="none" width="1569" height="100" x="0" y="500"></rect></g></g><g transform="translate(0,-8385)"><g><rect fill="black" stroke="none" width="1569" height="100" x="0" y="-500"></rect></g></g></g><g transform="translate(1551,0)"><g transform="translate(0,8584)"><g><rect fill="black" stroke="none" width="41597" height="100" x="0" y="500"></rect></g></g><g transform="translate(0,7284)"><use xlink:href="#E285-MJMAIN-31"></use><use xlink:href="#E285-MJMAIN-2E" x="500" y="0"></use><g transform="translate(778,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(1608,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">初</text></g><g transform="translate(2439,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">始</text></g><g transform="translate(3269,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">化</text></g><g transform="translate(4100,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(4931,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">任</text></g><g transform="translate(5761,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">意</text></g><g transform="translate(6592,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">初</text></g><g transform="translate(7423,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">始</text></g><g transform="translate(8253,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">化</text></g><g transform="translate(9084,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">评</text></g><g transform="translate(9915,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">估</text></g><g transform="translate(10745,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">网</text></g><g transform="translate(11576,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">络</text></g><g transform="translate(12657,0)"><use xlink:href="#E285-MJMATHI-71" x="0" y="0"></use><use xlink:href="#E285-MJMAIN-28" x="460" y="0"></use><use xlink:href="#E285-MJMAIN-22C5" x="849" y="0"></use><use xlink:href="#E285-MJMAIN-2C" x="1127" y="0"></use><use xlink:href="#E285-MJMAIN-22C5" x="1571" y="0"></use><use xlink:href="#E285-MJMAIN-3B" x="1849" y="0"></use><use xlink:href="#E285-MJMAINB-77" x="2294" y="0"></use><use xlink:href="#E285-MJMAIN-29" x="3125" y="0"></use></g><g transform="translate(16171,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">的</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">参</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">数</text></g></g><use xlink:href="#E285-MJMAINB-77" x="19163" y="0"></use><g transform="translate(19994,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">；</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">目</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">标</text></g><g transform="translate(2741,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">网</text></g><g transform="translate(3572,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">络</text></g></g><g transform="translate(24647,0)"><use xlink:href="#E285-MJMATHI-71" x="0" y="0"></use><use xlink:href="#E285-MJMAIN-28" x="460" y="0"></use><use xlink:href="#E285-MJMAIN-22C5" x="849" y="0"></use><use xlink:href="#E285-MJMAIN-2C" x="1127" y="0"></use><use xlink:href="#E285-MJMAIN-22C5" x="1571" y="0"></use><use xlink:href="#E285-MJMAIN-3B" x="1849" y="0"></use><g transform="translate(2294,0)"><use xlink:href="#E285-MJMAINB-77" x="0" y="0"></use><g transform="translate(831,-150)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(29.368) matrix(1 0 0 -1 0 0)">目</text><g transform="translate(587,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(29.368) matrix(1 0 0 -1 0 0)">标</text></g></g></g><use xlink:href="#E285-MJMAIN-29" x="4400" y="0"></use></g><g transform="translate(29436,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">的</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">参</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">数</text></g></g><g transform="translate(32428,0)"><use xlink:href="#E285-MJMAINB-77" x="0" y="0"></use><g transform="translate(831,-150)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(29.368) matrix(1 0 0 -1 0 0)">目</text><g transform="translate(587,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(29.368) matrix(1 0 0 -1 0 0)">标</text></g></g><use xlink:href="#E285-MJMAIN-2190" x="2383" y="0"></use><use xlink:href="#E285-MJMAINB-77" x="3661" y="0"></use></g><g transform="translate(36920,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">。</text></g></g></g><g transform="translate(0,5934)"><use xlink:href="#E285-MJMAIN-22EF" x="166" y="0"></use><g transform="translate(2505,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">同</text><g transform="translate(830,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">算</text></g><g transform="translate(1661,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">法</text></g><use xlink:href="#E285-MJMAIN-36" x="2741" y="0"></use><use xlink:href="#E285-MJMAIN-2D" x="3241" y="0"></use><use xlink:href="#E285-MJMAIN-38" x="3574" y="0"></use></g><use xlink:href="#E285-MJMAIN-22EF" x="7746" y="0"></use></g><g transform="translate(0,4625)"><g transform="translate(4000,0)"><use xlink:href="#E285-MJMAIN-32"></use><use xlink:href="#E285-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E285-MJMAIN-32" x="778" y="0"></use><use xlink:href="#E285-MJMAIN-2E" x="1278" y="0"></use><use xlink:href="#E285-MJMAIN-32" x="1556" y="0"></use><g transform="translate(2056,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(2886,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">存</text></g><g transform="translate(3717,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">储</text></g><g transform="translate(4547,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(5378,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">将</text></g><g transform="translate(6209,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">经</text></g><g transform="translate(7039,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">验</text></g><g transform="translate(8120,0)"><use xlink:href="#E285-MJMAIN-28" x="0" y="0"></use><use xlink:href="#E285-MJMATHI-53" x="389" y="0"></use><use xlink:href="#E285-MJMAIN-2C" x="1034" y="0"></use><use xlink:href="#E285-MJMATHI-41" x="1478" y="0"></use><use xlink:href="#E285-MJMAIN-2C" x="2228" y="0"></use><use xlink:href="#E285-MJMATHI-52" x="2673" y="0"></use><use xlink:href="#E285-MJMAIN-2C" x="3432" y="0"></use><g transform="translate(3877,0)"><use xlink:href="#E285-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E285-MJMAIN-2032" x="925" y="583"></use></g><use xlink:href="#E285-MJMAIN-29" x="4826" y="0"></use></g><g transform="translate(13335,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">存</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">入</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">经</text></g><g transform="translate(2741,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">验</text></g><g transform="translate(3572,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">库</text></g></g><use xlink:href="#E285-MJCAL-44" x="17988" y="0"></use><g transform="translate(18759,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">中</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">；</text></g></g></g></g><g transform="translate(0,3275)"><g transform="translate(4000,0)"><use xlink:href="#E285-MJMAIN-32"></use><use xlink:href="#E285-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E285-MJMAIN-32" x="778" y="0"></use><use xlink:href="#E285-MJMAIN-2E" x="1278" y="0"></use><use xlink:href="#E285-MJMAIN-33" x="1556" y="0"></use><g transform="translate(2056,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(2886,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">回</text></g><g transform="translate(3717,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">放</text></g><g transform="translate(4547,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(5378,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">从</text></g><g transform="translate(6209,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">经</text></g><g transform="translate(7039,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">验</text></g><g transform="translate(7870,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">库</text></g><use xlink:href="#E285-MJCAL-44" x="8951" y="0"></use><g transform="translate(9722,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">中</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">选</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">取</text></g><g transform="translate(2741,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">一</text></g><g transform="translate(3572,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">批</text></g><g transform="translate(4403,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">经</text></g><g transform="translate(5233,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">验</text></g></g><g transform="translate(16036,0)"><use xlink:href="#E285-MJMAIN-28" x="0" y="0"></use><g transform="translate(389,0)"><use xlink:href="#E285-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E285-MJMATHI-69" x="866" y="-213"></use></g><use xlink:href="#E285-MJMAIN-2C" x="1345" y="0"></use><g transform="translate(1790,0)"><use xlink:href="#E285-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E285-MJMATHI-69" x="1060" y="-213"></use></g><use xlink:href="#E285-MJMAIN-2C" x="2884" y="0"></use><g transform="translate(3329,0)"><use xlink:href="#E285-MJMATHI-52" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E285-MJMATHI-69" x="1073" y="-213"></use></g><use xlink:href="#E285-MJMAIN-2C" x="4432" y="0"></use><g transform="translate(4876,0)"><use xlink:href="#E285-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E285-MJMAIN-2032" x="925" y="444"></use><use transform="scale(0.707)" xlink:href="#E285-MJMATHI-69" x="866" y="-429"></use></g><use xlink:href="#E285-MJMAIN-29" x="5833" y="0"></use><use xlink:href="#E285-MJMAIN-2C" x="6222" y="0"></use><use xlink:href="#E285-MJMATHI-69" x="6945" y="0"></use><use xlink:href="#E285-MJMAIN-2208" x="7568" y="0"></use><use xlink:href="#E285-MJCAL-42" x="8512" y="0"></use></g><g transform="translate(25213,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">；</text></g></g></g></g><g transform="translate(0,1864)"><g transform="translate(4000,0)"><use xlink:href="#E285-MJMAIN-32"></use><use xlink:href="#E285-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E285-MJMAIN-32" x="778" y="0"></use><use xlink:href="#E285-MJMAIN-2E" x="1278" y="0"></use><use xlink:href="#E285-MJMAIN-34" x="1556" y="0"></use><g transform="translate(2056,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(2886,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">计</text></g><g transform="translate(3717,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">算</text></g><g transform="translate(4547,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">回</text></g><g transform="translate(5378,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">报</text></g><g transform="translate(6209,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">的</text></g><g transform="translate(7039,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">估</text></g><g transform="translate(7870,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">计</text></g><g transform="translate(8701,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">值</text></g><g transform="translate(9531,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(10362,0)"><use xlink:href="#E285-MJMATHI-55" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E285-MJMATHI-69" x="965" y="-213"></use><use xlink:href="#E285-MJMAIN-2190" x="1304" y="0"></use><g transform="translate(2582,0)"><use xlink:href="#E285-MJMATHI-52" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E285-MJMATHI-69" x="1073" y="-213"></use></g><use xlink:href="#E285-MJMAIN-2B" x="3907" y="0"></use><use xlink:href="#E285-MJMATHI-3B3" x="4907" y="0"></use><g transform="translate(5617,0)"><use xlink:href="#E285-MJMAIN-6D"></use><use xlink:href="#E285-MJMAIN-61" x="833" y="0"></use><use xlink:href="#E285-MJMAIN-78" x="1333" y="0"></use><use transform="scale(0.707)" xlink:href="#E285-MJMATHI-61" x="1051" y="-865"></use></g><use xlink:href="#E285-MJMATHI-71" x="7811" y="0"></use><use xlink:href="#E285-MJMAIN-28" x="8271" y="0"></use><g transform="translate(8660,0)"><use xlink:href="#E285-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E285-MJMAIN-2032" x="925" y="444"></use><use transform="scale(0.707)" xlink:href="#E285-MJMATHI-69" x="866" y="-429"></use></g><use xlink:href="#E285-MJMAIN-2C" x="9617" y="0"></use><use xlink:href="#E285-MJMATHI-61" x="10062" y="0"></use><use xlink:href="#E285-MJMAIN-3B" x="10591" y="0"></use><g transform="translate(11036,0)"><use xlink:href="#E285-MJMAINB-77" x="0" y="0"></use><g transform="translate(831,-150)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(29.368) matrix(1 0 0 -1 0 0)">目</text><g transform="translate(587,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(29.368) matrix(1 0 0 -1 0 0)">标</text></g></g></g><use xlink:href="#E285-MJMAIN-29" x="13141" y="0"></use><use xlink:href="#E285-MJMAIN-2C" x="13530" y="0"></use><use xlink:href="#E285-MJMATHI-69" x="14253" y="0"></use><use xlink:href="#E285-MJMAIN-2208" x="14876" y="0"></use><use xlink:href="#E285-MJCAL-42" x="15820" y="0"></use></g><g transform="translate(26847,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">；</text></g></g></g></g><g transform="translate(0,-497)"><g transform="translate(4000,0)"><use xlink:href="#E285-MJMAIN-32"></use><use xlink:href="#E285-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E285-MJMAIN-32" x="778" y="0"></use><use xlink:href="#E285-MJMAIN-2E" x="1278" y="0"></use><use xlink:href="#E285-MJMAIN-35" x="1556" y="0"></use><g transform="translate(2306,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(3136,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">更</text></g><g transform="translate(3967,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">新</text></g><g transform="translate(4797,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">动</text></g><g transform="translate(5628,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">作</text></g><g transform="translate(6459,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">价</text></g><g transform="translate(7289,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">值</text></g><g transform="translate(8120,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">函</text></g><g transform="translate(8951,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">数</text></g><g transform="translate(9781,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(10612,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">更</text></g><g transform="translate(11443,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">新</text></g><use xlink:href="#E285-MJMAINB-77" x="12523" y="0"></use><g transform="translate(13354,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">以</text></g><g transform="translate(1080,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">减</text></g><g transform="translate(1911,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">小</text></g></g><g transform="translate(16346,0)"><g transform="translate(120,0)"><rect stroke="none" width="1340" height="60" x="0" y="220"></rect><use xlink:href="#E285-MJMAIN-31" x="420" y="676"></use><g transform="translate(60,-694)"><use xlink:href="#E285-MJMAIN-7C" x="0" y="0"></use><use xlink:href="#E285-MJCAL-42" x="278" y="0"></use><use xlink:href="#E285-MJMAIN-7C" x="942" y="0"></use></g></g><g transform="translate(1746,0)"><use xlink:href="#E285-MJSZ2-2211" x="0" y="0"></use><g transform="translate(129,-1116)"><use transform="scale(0.707)" xlink:href="#E285-MJMATHI-69" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E285-MJMAIN-2208" x="345" y="0"></use><use transform="scale(0.707)" xlink:href="#E285-MJCAL-42" x="1012" y="0"></use></g></g><use xlink:href="#E285-MJMAIN-5B" x="3190" y="0"></use><g transform="translate(3468,0)"><use xlink:href="#E285-MJMATHI-55" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E285-MJMATHI-69" x="965" y="-213"></use></g><use xlink:href="#E285-MJMAIN-2212" x="4717" y="0"></use><use xlink:href="#E285-MJMATHI-71" x="5718" y="0"></use><use xlink:href="#E285-MJMAIN-28" x="6178" y="0"></use><g transform="translate(6567,0)"><use xlink:href="#E285-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E285-MJMATHI-69" x="866" y="-213"></use></g><use xlink:href="#E285-MJMAIN-2C" x="7524" y="0"></use><g transform="translate(7968,0)"><use xlink:href="#E285-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E285-MJMATHI-69" x="1060" y="-213"></use></g><use xlink:href="#E285-MJMAIN-3B" x="9062" y="0"></use><use xlink:href="#E285-MJMAINB-77" x="9507" y="0"></use><use xlink:href="#E285-MJMAIN-29" x="10338" y="0"></use><g transform="translate(10727,0)"><use xlink:href="#E285-MJMAIN-5D" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E285-MJMAIN-32" x="393" y="583"></use></g></g><g transform="translate(27805,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">，</text></g></g></g></g><g transform="translate(0,-3383)"><g transform="translate(6444,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">如</text><g transform="translate(1080,0)"><use xlink:href="#E285-MJMAINB-77" x="0" y="0"></use><use xlink:href="#E285-MJMAIN-2190" x="1108" y="0"></use><use xlink:href="#E285-MJMAINB-77" x="2386" y="0"></use><use xlink:href="#E285-MJMAIN-2B" x="3439" y="0"></use><use xlink:href="#E285-MJMATHI-3B1" x="4440" y="0"></use><g transform="translate(5080,0)"><g transform="translate(120,0)"><rect stroke="none" width="1340" height="60" x="0" y="220"></rect><use xlink:href="#E285-MJMAIN-31" x="420" y="676"></use><g transform="translate(60,-694)"><use xlink:href="#E285-MJMAIN-7C" x="0" y="0"></use><use xlink:href="#E285-MJCAL-42" x="278" y="0"></use><use xlink:href="#E285-MJMAIN-7C" x="942" y="0"></use></g></g></g><g transform="translate(6826,0)"><use xlink:href="#E285-MJSZ2-2211" x="0" y="0"></use><g transform="translate(129,-1116)"><use transform="scale(0.707)" xlink:href="#E285-MJMATHI-69" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E285-MJMAIN-2208" x="345" y="0"></use><use transform="scale(0.707)" xlink:href="#E285-MJCAL-42" x="1012" y="0"></use></g></g><use xlink:href="#E285-MJMAIN-5B" x="8270" y="0"></use><g transform="translate(8548,0)"><use xlink:href="#E285-MJMATHI-55" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E285-MJMATHI-69" x="965" y="-213"></use></g><use xlink:href="#E285-MJMAIN-2212" x="9797" y="0"></use><use xlink:href="#E285-MJMATHI-71" x="10798" y="0"></use><use xlink:href="#E285-MJMAIN-28" x="11258" y="0"></use><g transform="translate(11647,0)"><use xlink:href="#E285-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E285-MJMATHI-69" x="866" y="-213"></use></g><use xlink:href="#E285-MJMAIN-2C" x="12604" y="0"></use><g transform="translate(13048,0)"><use xlink:href="#E285-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E285-MJMATHI-69" x="1060" y="-213"></use></g><use xlink:href="#E285-MJMAIN-3B" x="14142" y="0"></use><use xlink:href="#E285-MJMAINB-77" x="14587" y="0"></use><use xlink:href="#E285-MJMAIN-29" x="15418" y="0"></use><use xlink:href="#E285-MJMAIN-5D" x="15807" y="0"></use><use xlink:href="#E285-MJMAIN-2207" x="16085" y="0"></use><use xlink:href="#E285-MJMATHI-71" x="16918" y="0"></use><use xlink:href="#E285-MJMAIN-28" x="17378" y="0"></use><g transform="translate(17767,0)"><use xlink:href="#E285-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E285-MJMATHI-69" x="866" y="-213"></use></g><use xlink:href="#E285-MJMAIN-2C" x="18724" y="0"></use><g transform="translate(19168,0)"><use xlink:href="#E285-MJMATHI-41" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E285-MJMATHI-69" x="1060" y="-213"></use></g><use xlink:href="#E285-MJMAIN-3B" x="20262" y="0"></use><use xlink:href="#E285-MJMAINB-77" x="20707" y="0"></use><use xlink:href="#E285-MJMAIN-29" x="21538" y="0"></use></g><g transform="translate(23008,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">；</text></g></g></g></g><g transform="translate(0,-5736)"><g transform="translate(4000,0)"><use xlink:href="#E285-MJMAIN-32"></use><use xlink:href="#E285-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E285-MJMAIN-32" x="778" y="0"></use><use xlink:href="#E285-MJMAIN-2E" x="1278" y="0"></use><use xlink:href="#E285-MJMAIN-36" x="1556" y="0"></use><g transform="translate(2306,0)"><use xlink:href="#E285-MJMATHI-53" x="444" y="0"></use><use xlink:href="#E285-MJMAIN-2190" x="1367" y="0"></use><g transform="translate(2645,0)"><use xlink:href="#E285-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E285-MJMAIN-2032" x="925" y="583"></use></g></g><g transform="translate(5900,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">；</text></g></g></g></g><g transform="translate(0,-7036)"><g transform="translate(4000,0)"><use xlink:href="#E285-MJMAIN-32"></use><use xlink:href="#E285-MJMAIN-2E" x="500" y="0"></use><use xlink:href="#E285-MJMAIN-32" x="778" y="0"></use><use xlink:href="#E285-MJMAIN-2E" x="1278" y="0"></use><use xlink:href="#E285-MJMAIN-37" x="1556" y="0"></use><g transform="translate(2056,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(2886,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">更</text></g><g transform="translate(3717,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">新</text></g><g transform="translate(4547,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">目</text></g><g transform="translate(5378,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">标</text></g><g transform="translate(6209,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">网</text></g><g transform="translate(7039,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">络</text></g><g transform="translate(7870,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(8701,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">在</text></g><g transform="translate(9531,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">一</text></g><g transform="translate(10362,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">定</text></g><g transform="translate(11193,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">条</text></g><g transform="translate(12023,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">件</text></g><g transform="translate(12854,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">下</text></g><g transform="translate(13685,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">（</text></g><g transform="translate(14515,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">例</text></g><g transform="translate(15346,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">如</text></g><g transform="translate(16177,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">访</text></g><g transform="translate(17007,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">问</text></g><g transform="translate(17838,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">本</text></g><g transform="translate(18669,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">步</text></g><g transform="translate(19499,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">若</text></g><g transform="translate(20330,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">干</text></g><g transform="translate(21160,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">次</text></g><g transform="translate(21991,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">）</text></g><g transform="translate(22822,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">更</text></g><g transform="translate(23652,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">新</text></g><g transform="translate(24483,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">目</text></g><g transform="translate(25314,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">标</text></g><g transform="translate(26144,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">网</text></g><g transform="translate(26975,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">络</text></g><g transform="translate(27806,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">的</text></g><g transform="translate(28636,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">权</text></g><g transform="translate(29467,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">重</text></g><g transform="translate(30548,0)"><use xlink:href="#E285-MJMAINB-77" x="0" y="0"></use><g transform="translate(831,-150)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(29.368) matrix(1 0 0 -1 0 0)">目</text><g transform="translate(587,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(29.368) matrix(1 0 0 -1 0 0)">标</text></g></g><use xlink:href="#E285-MJMAIN-2190" x="2383" y="0"></use><use xlink:href="#E285-MJMAINB-77" x="3661" y="0"></use></g><g transform="translate(35040,0)"><g transform="translate(250,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(41.533) matrix(1 0 0 -1 0 0)">。</text></g></g></g></g><g transform="translate(0,-8385)"><g><rect fill="black" stroke="none" width="41597" height="100" x="0" y="-500"></rect></g></g></g></g></g></svg></span></div><script type="math/tex; mode=display" id="MathJax-Element-258">\; \\ \; \\
\large \textbf{算法 6-9   带经验回放和目标网络的深度 Q 学习最优策略求解} \\
\begin{split}
\rule[5pt]{10mm}{0.1em} &\rule[5pt]{265mm}{0.1em} \\
&\text{1.（初始化）任意初始化评估网络 $q(\cdot,\cdot;\bold w)$ 的参数 $\bold w$ ；目标网络 $q(\cdot,\cdot;\bold w_{目标})$ 的参数 $\bold w_{目标} \leftarrow \bold w$ 。} \\
&\cdots \quad \text{同算法 6-8} \quad \cdots \\
&\qquad \qquad \text{2.2.2（存储）将经验 $(S,A,R,S')$ 存入经验库 $\mathcal D$ 中；} \\
&\qquad \qquad \text{2.2.3（回放）从经验库 $\mathcal D$ 中选取一批经验 $(S_i,A_i,R_i,S_i'),\; i \in \mathcal B$ ；} \\
&\qquad \qquad \text{2.2.4（计算回报的估计值）$U_i \leftarrow R_i + \gamma \max_a\, q(S_i',a;\bold w_{目标}),\; i \in \mathcal B$ ；} \\
&\qquad \qquad \text{2.2.5 （更新动作价值函数）更新 $\bold w$ 以减小 $\frac{1}{|\mathcal B|} \sum_{i \in \mathcal B} [U_i-q(S_i,A_i;\bold w)]^2$ ，} \\
&\qquad \qquad \qquad \;\, \text{如 $\bold w \leftarrow \bold w + \alpha\frac{1}{|\mathcal B|} \sum_{i \in \mathcal B}  [U_i - q(S_i,A_i;\bold w)]\nabla q(S_i,A_i;\bold w)$ ；} \\
&\qquad \qquad \text{2.2.6 $\;\, S \leftarrow S'$ ；}\\
&\qquad \qquad \text{2.2.7（更新目标网络）在一定条件下（例如访问本步若干次）更新目标网络的权重 $\bold w_{目标} \leftarrow \bold w$ 。}\\
\rule[-5pt]{10mm}{0.1em} &\rule[-5pt]{265mm}{0.1em}
\end{split}
\; \\ \; \\</script></div></div><p><span>在更新目标网络时，还可以引入一个学习率 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="4.447ex" height="1.842ex" viewBox="0 -500.4 1914.7 793.1" role="img" focusable="false" style="vertical-align: -0.68ex;"><defs><path stroke-width="0" id="E342-MJMATHI-3B1" d="M34 156Q34 270 120 356T309 442Q379 442 421 402T478 304Q484 275 485 237V208Q534 282 560 374Q564 388 566 390T582 393Q603 393 603 385Q603 376 594 346T558 261T497 161L486 147L487 123Q489 67 495 47T514 26Q528 28 540 37T557 60Q559 67 562 68T577 70Q597 70 597 62Q597 56 591 43Q579 19 556 5T512 -10H505Q438 -10 414 62L411 69L400 61Q390 53 370 41T325 18T267 -2T203 -11Q124 -11 79 39T34 156ZM208 26Q257 26 306 47T379 90L403 112Q401 255 396 290Q382 405 304 405Q235 405 183 332Q156 292 139 224T121 120Q121 71 146 49T208 26Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E342-MJMATHI-3B1" x="0" y="0"></use><g transform="translate(640,-150)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(29.368) matrix(1 0 0 -1 0 0)">目</text><g transform="translate(587,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(29.368) matrix(1 0 0 -1 0 0)">标</text></g></g></g></svg></span><script type="math/tex">\alpha_{目标}</script><span> ，然后使用加权平均更新：</span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="32.865ex" height="2.71ex" viewBox="0 -832.7 14150.3 1166.9" role="img" focusable="false" style="vertical-align: -0.776ex;"><defs><path stroke-width="0" id="E343-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E343-MJMAIN-2190" d="M944 261T944 250T929 230H165Q167 228 182 216T211 189T244 152T277 96T303 25Q308 7 308 0Q308 -11 288 -11Q281 -11 278 -11T272 -7T267 2T263 21Q245 94 195 151T73 236Q58 242 55 247Q55 254 59 257T73 264Q121 283 158 314T215 375T247 434T264 480L267 497Q269 503 270 505T275 509T288 511Q308 511 308 500Q308 493 303 475Q293 438 278 406T246 352T215 315T185 287T165 270H929Q944 261 944 250Z"></path><path stroke-width="0" id="E343-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E343-MJMAIN-31" d="M213 578L200 573Q186 568 160 563T102 556H83V602H102Q149 604 189 617T245 641T273 663Q275 666 285 666Q294 666 302 660V361L303 61Q310 54 315 52T339 48T401 46H427V0H416Q395 3 257 3Q121 3 100 0H88V46H114Q136 46 152 46T177 47T193 50T201 52T207 57T213 61V578Z"></path><path stroke-width="0" id="E343-MJMAIN-2212" d="M84 237T84 250T98 270H679Q694 262 694 250T679 230H98Q84 237 84 250Z"></path><path stroke-width="0" id="E343-MJMATHI-3B1" d="M34 156Q34 270 120 356T309 442Q379 442 421 402T478 304Q484 275 485 237V208Q534 282 560 374Q564 388 566 390T582 393Q603 393 603 385Q603 376 594 346T558 261T497 161L486 147L487 123Q489 67 495 47T514 26Q528 28 540 37T557 60Q559 67 562 68T577 70Q597 70 597 62Q597 56 591 43Q579 19 556 5T512 -10H505Q438 -10 414 62L411 69L400 61Q390 53 370 41T325 18T267 -2T203 -11Q124 -11 79 39T34 156ZM208 26Q257 26 306 47T379 90L403 112Q401 255 396 290Q382 405 304 405Q235 405 183 332Q156 292 139 224T121 120Q121 71 146 49T208 26Z"></path><path stroke-width="0" id="E343-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E343-MJMAIN-2B" d="M56 237T56 250T70 270H369V420L370 570Q380 583 389 583Q402 583 409 568V270H707Q722 262 722 250T707 230H409V-68Q401 -82 391 -82H389H387Q375 -82 369 -68V230H70Q56 237 56 250Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E343-MJMAINB-77" x="0" y="0"></use><g transform="translate(831,-150)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(29.368) matrix(1 0 0 -1 0 0)">目</text><g transform="translate(587,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(29.368) matrix(1 0 0 -1 0 0)">标</text></g></g><use xlink:href="#E343-MJMAIN-2190" x="2383" y="0"></use><use xlink:href="#E343-MJMAIN-28" x="3661" y="0"></use><use xlink:href="#E343-MJMAIN-31" x="4050" y="0"></use><use xlink:href="#E343-MJMAIN-2212" x="4772" y="0"></use><g transform="translate(5772,0)"><use xlink:href="#E343-MJMATHI-3B1" x="0" y="0"></use><g transform="translate(640,-150)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(29.368) matrix(1 0 0 -1 0 0)">目</text><g transform="translate(587,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(29.368) matrix(1 0 0 -1 0 0)">标</text></g></g></g><use xlink:href="#E343-MJMAIN-29" x="7687" y="0"></use><g transform="translate(8076,0)"><use xlink:href="#E343-MJMAINB-77" x="0" y="0"></use><g transform="translate(831,-150)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(29.368) matrix(1 0 0 -1 0 0)">目</text><g transform="translate(587,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(29.368) matrix(1 0 0 -1 0 0)">标</text></g></g></g><use xlink:href="#E343-MJMAIN-2B" x="10404" y="0"></use><g transform="translate(11404,0)"><use xlink:href="#E343-MJMATHI-3B1" x="0" y="0"></use><g transform="translate(640,-150)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(29.368) matrix(1 0 0 -1 0 0)">目</text><g transform="translate(587,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(29.368) matrix(1 0 0 -1 0 0)">标</text></g></g></g><use xlink:href="#E343-MJMAINB-77" x="13319" y="0"></use></g></svg></span><script type="math/tex">\bold w_{目标} \leftarrow (1-\alpha_{目标})\bold w_{目标} + \alpha_{目标}\bold w</script><span> 。对于分布式学习的情况，有很多独立拷贝（worker）同时会修改目标网络，则就更常用学习率 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="12.449ex" height="2.71ex" viewBox="0 -832.7 5359.9 1166.9" role="img" focusable="false" style="vertical-align: -0.776ex;"><defs><path stroke-width="0" id="E344-MJMATHI-3B1" d="M34 156Q34 270 120 356T309 442Q379 442 421 402T478 304Q484 275 485 237V208Q534 282 560 374Q564 388 566 390T582 393Q603 393 603 385Q603 376 594 346T558 261T497 161L486 147L487 123Q489 67 495 47T514 26Q528 28 540 37T557 60Q559 67 562 68T577 70Q597 70 597 62Q597 56 591 43Q579 19 556 5T512 -10H505Q438 -10 414 62L411 69L400 61Q390 53 370 41T325 18T267 -2T203 -11Q124 -11 79 39T34 156ZM208 26Q257 26 306 47T379 90L403 112Q401 255 396 290Q382 405 304 405Q235 405 183 332Q156 292 139 224T121 120Q121 71 146 49T208 26Z"></path><path stroke-width="0" id="E344-MJMAIN-2208" d="M84 250Q84 372 166 450T360 539Q361 539 377 539T419 540T469 540H568Q583 532 583 520Q583 511 570 501L466 500Q355 499 329 494Q280 482 242 458T183 409T147 354T129 306T124 272V270H568Q583 262 583 250T568 230H124V228Q124 207 134 177T167 112T231 48T328 7Q355 1 466 0H570Q583 -10 583 -20Q583 -32 568 -40H471Q464 -40 446 -40T417 -41Q262 -41 172 45Q84 127 84 250Z"></path><path stroke-width="0" id="E344-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E344-MJMAIN-30" d="M96 585Q152 666 249 666Q297 666 345 640T423 548Q460 465 460 320Q460 165 417 83Q397 41 362 16T301 -15T250 -22Q224 -22 198 -16T137 16T82 83Q39 165 39 320Q39 494 96 585ZM321 597Q291 629 250 629Q208 629 178 597Q153 571 145 525T137 333Q137 175 145 125T181 46Q209 16 250 16Q290 16 318 46Q347 76 354 130T362 333Q362 478 354 524T321 597Z"></path><path stroke-width="0" id="E344-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E344-MJMAIN-31" d="M213 578L200 573Q186 568 160 563T102 556H83V602H102Q149 604 189 617T245 641T273 663Q275 666 285 666Q294 666 302 660V361L303 61Q310 54 315 52T339 48T401 46H427V0H416Q395 3 257 3Q121 3 100 0H88V46H114Q136 46 152 46T177 47T193 50T201 52T207 57T213 61V578Z"></path><path stroke-width="0" id="E344-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E344-MJMATHI-3B1" x="0" y="0"></use><g transform="translate(640,-150)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(29.368) matrix(1 0 0 -1 0 0)">目</text><g transform="translate(587,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(29.368) matrix(1 0 0 -1 0 0)">标</text></g></g><use xlink:href="#E344-MJMAIN-2208" x="2192" y="0"></use><use xlink:href="#E344-MJMAIN-28" x="3137" y="0"></use><use xlink:href="#E344-MJMAIN-30" x="3526" y="0"></use><use xlink:href="#E344-MJMAIN-2C" x="4026" y="0"></use><use xlink:href="#E344-MJMAIN-31" x="4470" y="0"></use><use xlink:href="#E344-MJMAIN-29" x="4970" y="0"></use></g></svg></span><script type="math/tex">\alpha_{目标} \in (0,1)</script><span> 。</span></p><p><span>Deepmind 于 2015 年发表论文《Deep reinforcement learning with double Q_learning》，将双重 Q 学习用于深度 Q 网络，得到了</span><strong><span>双重深度 Q 网络</span></strong><span>（Double Deep Q Network, Double DQN）。由于深度 Q 网络已经有了评估网络和目标网络，所以双重深度 Q 学习在估计回报时只需要用评估网络确定动作，用目标网络确定回报的估计即可，那么将算法 6-9 中的计算回报的估计值部分改为：</span></p><div contenteditable="false" spellcheck="false" class="mathjax-block md-end-block md-math-block md-rawblock" id="mathjax-n99" cid="n99" mdtype="math_block"><div class="md-rawblock-container md-math-container" tabindex="-1"><div class="MathJax_SVG_Display"><span class="MathJax_SVG" id="MathJax-Element-259-Frame" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="98.296ex" height="4.254ex" viewBox="0 -1164.9 42321.7 1831.4" role="img" focusable="false" style="vertical-align: -1.548ex; max-width: 100%;"><defs><path stroke-width="0" id="E286-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E286-MJMAIN-31" d="M213 578L200 573Q186 568 160 563T102 556H83V602H102Q149 604 189 617T245 641T273 663Q275 666 285 666Q294 666 302 660V361L303 61Q310 54 315 52T339 48T401 46H427V0H416Q395 3 257 3Q121 3 100 0H88V46H114Q136 46 152 46T177 47T193 50T201 52T207 57T213 61V578Z"></path><path stroke-width="0" id="E286-MJMAIN-32" d="M109 429Q82 429 66 447T50 491Q50 562 103 614T235 666Q326 666 387 610T449 465Q449 422 429 383T381 315T301 241Q265 210 201 149L142 93L218 92Q375 92 385 97Q392 99 409 186V189H449V186Q448 183 436 95T421 3V0H50V19V31Q50 38 56 46T86 81Q115 113 136 137Q145 147 170 174T204 211T233 244T261 278T284 308T305 340T320 369T333 401T340 431T343 464Q343 527 309 573T212 619Q179 619 154 602T119 569T109 550Q109 549 114 549Q132 549 151 535T170 489Q170 464 154 447T109 429Z"></path><path stroke-width="0" id="E286-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E286-MJMATHI-55" d="M107 637Q73 637 71 641Q70 643 70 649Q70 673 81 682Q83 683 98 683Q139 681 234 681Q268 681 297 681T342 682T362 682Q378 682 378 672Q378 670 376 658Q371 641 366 638H364Q362 638 359 638T352 638T343 637T334 637Q295 636 284 634T266 623Q265 621 238 518T184 302T154 169Q152 155 152 140Q152 86 183 55T269 24Q336 24 403 69T501 205L552 406Q599 598 599 606Q599 633 535 637Q511 637 511 648Q511 650 513 660Q517 676 519 679T529 683Q532 683 561 682T645 680Q696 680 723 681T752 682Q767 682 767 672Q767 650 759 642Q756 637 737 637Q666 633 648 597Q646 592 598 404Q557 235 548 205Q515 105 433 42T263 -22Q171 -22 116 34T60 167V183Q60 201 115 421Q164 622 164 628Q164 635 107 637Z"></path><path stroke-width="0" id="E286-MJMATHI-69" d="M184 600Q184 624 203 642T247 661Q265 661 277 649T290 619Q290 596 270 577T226 557Q211 557 198 567T184 600ZM21 287Q21 295 30 318T54 369T98 420T158 442Q197 442 223 419T250 357Q250 340 236 301T196 196T154 83Q149 61 149 51Q149 26 166 26Q175 26 185 29T208 43T235 78T260 137Q263 149 265 151T282 153Q302 153 302 143Q302 135 293 112T268 61T223 11T161 -11Q129 -11 102 10T74 74Q74 91 79 106T122 220Q160 321 166 341T173 380Q173 404 156 404H154Q124 404 99 371T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Z"></path><path stroke-width="0" id="E286-MJMAIN-2190" d="M944 261T944 250T929 230H165Q167 228 182 216T211 189T244 152T277 96T303 25Q308 7 308 0Q308 -11 288 -11Q281 -11 278 -11T272 -7T267 2T263 21Q245 94 195 151T73 236Q58 242 55 247Q55 254 59 257T73 264Q121 283 158 314T215 375T247 434T264 480L267 497Q269 503 270 505T275 509T288 511Q308 511 308 500Q308 493 303 475Q293 438 278 406T246 352T215 315T185 287T165 270H929Q944 261 944 250Z"></path><path stroke-width="0" id="E286-MJMATHI-52" d="M230 637Q203 637 198 638T193 649Q193 676 204 682Q206 683 378 683Q550 682 564 680Q620 672 658 652T712 606T733 563T739 529Q739 484 710 445T643 385T576 351T538 338L545 333Q612 295 612 223Q612 212 607 162T602 80V71Q602 53 603 43T614 25T640 16Q668 16 686 38T712 85Q717 99 720 102T735 105Q755 105 755 93Q755 75 731 36Q693 -21 641 -21H632Q571 -21 531 4T487 82Q487 109 502 166T517 239Q517 290 474 313Q459 320 449 321T378 323H309L277 193Q244 61 244 59Q244 55 245 54T252 50T269 48T302 46H333Q339 38 339 37T336 19Q332 6 326 0H311Q275 2 180 2Q146 2 117 2T71 2T50 1Q33 1 33 10Q33 12 36 24Q41 43 46 45Q50 46 61 46H67Q94 46 127 49Q141 52 146 61Q149 65 218 339T287 628Q287 635 230 637ZM630 554Q630 586 609 608T523 636Q521 636 500 636T462 637H440Q393 637 386 627Q385 624 352 494T319 361Q319 360 388 360Q466 361 492 367Q556 377 592 426Q608 449 619 486T630 554Z"></path><path stroke-width="0" id="E286-MJMAIN-2B" d="M56 237T56 250T70 270H369V420L370 570Q380 583 389 583Q402 583 409 568V270H707Q722 262 722 250T707 230H409V-68Q401 -82 391 -82H389H387Q375 -82 369 -68V230H70Q56 237 56 250Z"></path><path stroke-width="0" id="E286-MJMATHI-3B3" d="M31 249Q11 249 11 258Q11 275 26 304T66 365T129 418T206 441Q233 441 239 440Q287 429 318 386T371 255Q385 195 385 170Q385 166 386 166L398 193Q418 244 443 300T486 391T508 430Q510 431 524 431H537Q543 425 543 422Q543 418 522 378T463 251T391 71Q385 55 378 6T357 -100Q341 -165 330 -190T303 -216Q286 -216 286 -188Q286 -138 340 32L346 51L347 69Q348 79 348 100Q348 257 291 317Q251 355 196 355Q148 355 108 329T51 260Q49 251 47 251Q45 249 31 249Z"></path><path stroke-width="0" id="E286-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E286-MJMATHI-53" d="M308 24Q367 24 416 76T466 197Q466 260 414 284Q308 311 278 321T236 341Q176 383 176 462Q176 523 208 573T273 648Q302 673 343 688T407 704H418H425Q521 704 564 640Q565 640 577 653T603 682T623 704Q624 704 627 704T632 705Q645 705 645 698T617 577T585 459T569 456Q549 456 549 465Q549 471 550 475Q550 478 551 494T553 520Q553 554 544 579T526 616T501 641Q465 662 419 662Q362 662 313 616T263 510Q263 480 278 458T319 427Q323 425 389 408T456 390Q490 379 522 342T554 242Q554 216 546 186Q541 164 528 137T492 78T426 18T332 -20Q320 -22 298 -22Q199 -22 144 33L134 44L106 13Q83 -14 78 -18T65 -22Q52 -22 52 -14Q52 -11 110 221Q112 227 130 227H143Q149 221 149 216Q149 214 148 207T144 186T142 153Q144 114 160 87T203 47T255 29T308 24Z"></path><path stroke-width="0" id="E286-MJMAIN-2032" d="M79 43Q73 43 52 49T30 61Q30 68 85 293T146 528Q161 560 198 560Q218 560 240 545T262 501Q262 496 260 486Q259 479 173 263T84 45T79 43Z"></path><path stroke-width="0" id="E286-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E286-MJMAIN-61" d="M137 305T115 305T78 320T63 359Q63 394 97 421T218 448Q291 448 336 416T396 340Q401 326 401 309T402 194V124Q402 76 407 58T428 40Q443 40 448 56T453 109V145H493V106Q492 66 490 59Q481 29 455 12T400 -6T353 12T329 54V58L327 55Q325 52 322 49T314 40T302 29T287 17T269 6T247 -2T221 -8T190 -11Q130 -11 82 20T34 107Q34 128 41 147T68 188T116 225T194 253T304 268H318V290Q318 324 312 340Q290 411 215 411Q197 411 181 410T156 406T148 403Q170 388 170 359Q170 334 154 320ZM126 106Q126 75 150 51T209 26Q247 26 276 49T315 109Q317 116 318 175Q318 233 317 233Q309 233 296 232T251 223T193 203T147 166T126 106Z"></path><path stroke-width="0" id="E286-MJMAIN-72" d="M36 46H50Q89 46 97 60V68Q97 77 97 91T98 122T98 161T98 203Q98 234 98 269T98 328L97 351Q94 370 83 376T38 385H20V408Q20 431 22 431L32 432Q42 433 60 434T96 436Q112 437 131 438T160 441T171 442H174V373Q213 441 271 441H277Q322 441 343 419T364 373Q364 352 351 337T313 322Q288 322 276 338T263 372Q263 381 265 388T270 400T273 405Q271 407 250 401Q234 393 226 386Q179 341 179 207V154Q179 141 179 127T179 101T180 81T180 66V61Q181 59 183 57T188 54T193 51T200 49T207 48T216 47T225 47T235 46T245 46H276V0H267Q249 3 140 3Q37 3 28 0H20V46H36Z"></path><path stroke-width="0" id="E286-MJMAIN-67" d="M329 409Q373 453 429 453Q459 453 472 434T485 396Q485 382 476 371T449 360Q416 360 412 390Q410 404 415 411Q415 412 416 414V415Q388 412 363 393Q355 388 355 386Q355 385 359 381T368 369T379 351T388 325T392 292Q392 230 343 187T222 143Q172 143 123 171Q112 153 112 133Q112 98 138 81Q147 75 155 75T227 73Q311 72 335 67Q396 58 431 26Q470 -13 470 -72Q470 -139 392 -175Q332 -206 250 -206Q167 -206 107 -175Q29 -140 29 -75Q29 -39 50 -15T92 18L103 24Q67 55 67 108Q67 155 96 193Q52 237 52 292Q52 355 102 398T223 442Q274 442 318 416L329 409ZM299 343Q294 371 273 387T221 404Q192 404 171 388T145 343Q142 326 142 292Q142 248 149 227T179 192Q196 182 222 182Q244 182 260 189T283 207T294 227T299 242Q302 258 302 292T299 343ZM403 -75Q403 -50 389 -34T348 -11T299 -2T245 0H218Q151 0 138 -6Q118 -15 107 -34T95 -74Q95 -84 101 -97T122 -127T170 -155T250 -167Q319 -167 361 -139T403 -75Z"></path><path stroke-width="0" id="E286-MJMAIN-6D" d="M41 46H55Q94 46 102 60V68Q102 77 102 91T102 122T103 161T103 203Q103 234 103 269T102 328V351Q99 370 88 376T43 385H25V408Q25 431 27 431L37 432Q47 433 65 434T102 436Q119 437 138 438T167 441T178 442H181V402Q181 364 182 364T187 369T199 384T218 402T247 421T285 437Q305 442 336 442Q351 442 364 440T387 434T406 426T421 417T432 406T441 395T448 384T452 374T455 366L457 361L460 365Q463 369 466 373T475 384T488 397T503 410T523 422T546 432T572 439T603 442Q729 442 740 329Q741 322 741 190V104Q741 66 743 59T754 49Q775 46 803 46H819V0H811L788 1Q764 2 737 2T699 3Q596 3 587 0H579V46H595Q656 46 656 62Q657 64 657 200Q656 335 655 343Q649 371 635 385T611 402T585 404Q540 404 506 370Q479 343 472 315T464 232V168V108Q464 78 465 68T468 55T477 49Q498 46 526 46H542V0H534L510 1Q487 2 460 2T422 3Q319 3 310 0H302V46H318Q379 46 379 62Q380 64 380 200Q379 335 378 343Q372 371 358 385T334 402T308 404Q263 404 229 370Q202 343 195 315T187 232V168V108Q187 78 188 68T191 55T200 49Q221 46 249 46H265V0H257L234 1Q210 2 183 2T145 3Q42 3 33 0H25V46H41Z"></path><path stroke-width="0" id="E286-MJMAIN-78" d="M201 0Q189 3 102 3Q26 3 17 0H11V46H25Q48 47 67 52T96 61T121 78T139 96T160 122T180 150L226 210L168 288Q159 301 149 315T133 336T122 351T113 363T107 370T100 376T94 379T88 381T80 383Q74 383 44 385H16V431H23Q59 429 126 429Q219 429 229 431H237V385Q201 381 201 369Q201 367 211 353T239 315T268 274L272 270L297 304Q329 345 329 358Q329 364 327 369T322 376T317 380T310 384L307 385H302V431H309Q324 428 408 428Q487 428 493 431H499V385H492Q443 385 411 368Q394 360 377 341T312 257L296 236L358 151Q424 61 429 57T446 50Q464 46 499 46H516V0H510H502Q494 1 482 1T457 2T432 2T414 3Q403 3 377 3T327 1L304 0H295V46H298Q309 46 320 51T331 63Q331 65 291 120L250 175Q249 174 219 133T185 88Q181 83 181 74Q181 63 188 55T206 46Q208 46 208 23V0H201Z"></path><path stroke-width="0" id="E286-MJMATHI-61" d="M33 157Q33 258 109 349T280 441Q331 441 370 392Q386 422 416 422Q429 422 439 414T449 394Q449 381 412 234T374 68Q374 43 381 35T402 26Q411 27 422 35Q443 55 463 131Q469 151 473 152Q475 153 483 153H487Q506 153 506 144Q506 138 501 117T481 63T449 13Q436 0 417 -8Q409 -10 393 -10Q359 -10 336 5T306 36L300 51Q299 52 296 50Q294 48 292 46Q233 -10 172 -10Q117 -10 75 30T33 157ZM351 328Q351 334 346 350T323 385T277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q217 26 254 59T298 110Q300 114 325 217T351 328Z"></path><path stroke-width="0" id="E286-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E286-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><g transform="translate(40543,0)"><g id="mjx-eqn-12" transform="translate(0,306)"><use xlink:href="#E286-MJMAIN-28"></use><use xlink:href="#E286-MJMAIN-31" x="389" y="0"></use><use xlink:href="#E286-MJMAIN-32" x="889" y="0"></use><use xlink:href="#E286-MJMAIN-29" x="1389" y="0"></use></g></g><g transform="translate(11945,0)"><g transform="translate(-19,0)"><g transform="translate(0,306)"><use xlink:href="#E286-MJMATHI-55" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E286-MJMATHI-69" x="965" y="-213"></use><use xlink:href="#E286-MJMAIN-2190" x="1304" y="0"></use><g transform="translate(2582,0)"><use xlink:href="#E286-MJMATHI-52" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E286-MJMATHI-69" x="1073" y="-213"></use></g><use xlink:href="#E286-MJMAIN-2B" x="3907" y="0"></use><use xlink:href="#E286-MJMATHI-3B3" x="4907" y="0"></use><use xlink:href="#E286-MJMATHI-71" x="5450" y="0"></use><use xlink:href="#E286-MJMAIN-28" x="5910" y="0"></use><g transform="translate(6299,0)"><use xlink:href="#E286-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E286-MJMAIN-2032" x="925" y="444"></use><use transform="scale(0.707)" xlink:href="#E286-MJMATHI-69" x="866" y="-429"></use></g><use xlink:href="#E286-MJMAIN-2C" x="7256" y="0"></use><g transform="translate(7701,0)"><use xlink:href="#E286-MJMAIN-61"></use><use xlink:href="#E286-MJMAIN-72" x="500" y="0"></use><use xlink:href="#E286-MJMAIN-67" x="892" y="0"></use><g transform="translate(1558,0)"><use xlink:href="#E286-MJMAIN-6D"></use><use xlink:href="#E286-MJMAIN-61" x="833" y="0"></use><use xlink:href="#E286-MJMAIN-78" x="1333" y="0"></use></g><use transform="scale(0.707)" xlink:href="#E286-MJMATHI-61" x="2153" y="-1140"></use></g><use xlink:href="#E286-MJMATHI-71" x="11398" y="0"></use><use xlink:href="#E286-MJMAIN-28" x="11858" y="0"></use><g transform="translate(12247,0)"><use xlink:href="#E286-MJMATHI-53" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E286-MJMAIN-2032" x="925" y="444"></use><use transform="scale(0.707)" xlink:href="#E286-MJMATHI-69" x="866" y="-429"></use></g><use xlink:href="#E286-MJMAIN-2C" x="13204" y="0"></use><use xlink:href="#E286-MJMATHI-61" x="13649" y="0"></use><use xlink:href="#E286-MJMAIN-3B" x="14178" y="0"></use><use xlink:href="#E286-MJMAINB-77" x="14623" y="0"></use><use xlink:href="#E286-MJMAIN-29" x="15454" y="0"></use><use xlink:href="#E286-MJMAIN-3B" x="15843" y="0"></use><g transform="translate(16287,0)"><use xlink:href="#E286-MJMAINB-77" x="0" y="0"></use><g transform="translate(831,-150)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(29.368) matrix(1 0 0 -1 0 0)">目</text><g transform="translate(587,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(29.368) matrix(1 0 0 -1 0 0)">标</text></g></g></g><use xlink:href="#E286-MJMAIN-29" x="18393" y="0"></use></g></g></g></g></svg></span></div><script type="math/tex; mode=display" id="MathJax-Element-259">U_i \leftarrow R_i + \gamma q(S_i',\underset{a}{\arg\max}\;q(S_i',a;\bold w);\bold w_{目标})</script></div></div><p><span>就得到了带经验回放的双重深度 Q 网络算法。</span></p><p><span>Z. Wang 等在 2015 年发表的论文《Dueling network architectures for deep reinforcement learning》提出了一种神经网络的结构——</span><strong><span>对偶网络</span></strong><span>（duel network）。对偶网络理论利用动作价值函数和状态价值函数之差定义了一个新的函数——</span><strong><span>优势函数</span></strong><span>（advantage function）：</span></p><div contenteditable="false" spellcheck="false" class="mathjax-block md-end-block md-math-block md-rawblock" id="mathjax-n102" cid="n102" mdtype="math_block"><div class="md-rawblock-container md-math-container" tabindex="-1"><div class="MathJax_SVG_Display"><span class="MathJax_SVG" id="MathJax-Element-260-Frame" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="98.296ex" height="2.71ex" viewBox="0 -832.7 42321.7 1166.9" role="img" focusable="false" style="vertical-align: -0.776ex; max-width: 100%;"><defs><path stroke-width="0" id="E287-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E287-MJMAIN-31" d="M213 578L200 573Q186 568 160 563T102 556H83V602H102Q149 604 189 617T245 641T273 663Q275 666 285 666Q294 666 302 660V361L303 61Q310 54 315 52T339 48T401 46H427V0H416Q395 3 257 3Q121 3 100 0H88V46H114Q136 46 152 46T177 47T193 50T201 52T207 57T213 61V578Z"></path><path stroke-width="0" id="E287-MJMAIN-33" d="M127 463Q100 463 85 480T69 524Q69 579 117 622T233 665Q268 665 277 664Q351 652 390 611T430 522Q430 470 396 421T302 350L299 348Q299 347 308 345T337 336T375 315Q457 262 457 175Q457 96 395 37T238 -22Q158 -22 100 21T42 130Q42 158 60 175T105 193Q133 193 151 175T169 130Q169 119 166 110T159 94T148 82T136 74T126 70T118 67L114 66Q165 21 238 21Q293 21 321 74Q338 107 338 175V195Q338 290 274 322Q259 328 213 329L171 330L168 332Q166 335 166 348Q166 366 174 366Q202 366 232 371Q266 376 294 413T322 525V533Q322 590 287 612Q265 626 240 626Q208 626 181 615T143 592T132 580H135Q138 579 143 578T153 573T165 566T175 555T183 540T186 520Q186 498 172 481T127 463Z"></path><path stroke-width="0" id="E287-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E287-MJMATHI-61" d="M33 157Q33 258 109 349T280 441Q331 441 370 392Q386 422 416 422Q429 422 439 414T449 394Q449 381 412 234T374 68Q374 43 381 35T402 26Q411 27 422 35Q443 55 463 131Q469 151 473 152Q475 153 483 153H487Q506 153 506 144Q506 138 501 117T481 63T449 13Q436 0 417 -8Q409 -10 393 -10Q359 -10 336 5T306 36L300 51Q299 52 296 50Q294 48 292 46Q233 -10 172 -10Q117 -10 75 30T33 157ZM351 328Q351 334 346 350T323 385T277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q217 26 254 59T298 110Q300 114 325 217T351 328Z"></path><path stroke-width="0" id="E287-MJMATHI-73" d="M131 289Q131 321 147 354T203 415T300 442Q362 442 390 415T419 355Q419 323 402 308T364 292Q351 292 340 300T328 326Q328 342 337 354T354 372T367 378Q368 378 368 379Q368 382 361 388T336 399T297 405Q249 405 227 379T204 326Q204 301 223 291T278 274T330 259Q396 230 396 163Q396 135 385 107T352 51T289 7T195 -10Q118 -10 86 19T53 87Q53 126 74 143T118 160Q133 160 146 151T160 120Q160 94 142 76T111 58Q109 57 108 57T107 55Q108 52 115 47T146 34T201 27Q237 27 263 38T301 66T318 97T323 122Q323 150 302 164T254 181T195 196T148 231Q131 256 131 289Z"></path><path stroke-width="0" id="E287-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E287-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E287-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E287-MJMAIN-2212" d="M84 237T84 250T98 270H679Q694 262 694 250T679 230H98Q84 237 84 250Z"></path><path stroke-width="0" id="E287-MJMATHI-76" d="M173 380Q173 405 154 405Q130 405 104 376T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Q21 294 29 316T53 368T97 419T160 441Q202 441 225 417T249 361Q249 344 246 335Q246 329 231 291T200 202T182 113Q182 86 187 69Q200 26 250 26Q287 26 319 60T369 139T398 222T409 277Q409 300 401 317T383 343T365 361T357 383Q357 405 376 424T417 443Q436 443 451 425T467 367Q467 340 455 284T418 159T347 40T241 -11Q177 -11 139 22Q102 54 102 117Q102 148 110 181T151 298Q173 362 173 380Z"></path><path stroke-width="0" id="E287-MJMAIN-2208" d="M84 250Q84 372 166 450T360 539Q361 539 377 539T419 540T469 540H568Q583 532 583 520Q583 511 570 501L466 500Q355 499 329 494Q280 482 242 458T183 409T147 354T129 306T124 272V270H568Q583 262 583 250T568 230H124V228Q124 207 134 177T167 112T231 48T328 7Q355 1 466 0H570Q583 -10 583 -20Q583 -32 568 -40H471Q464 -40 446 -40T417 -41Q262 -41 172 45Q84 127 84 250Z"></path><path stroke-width="0" id="E287-MJCAL-53" d="M554 512Q536 512 536 522Q536 525 539 539T542 564Q542 588 528 604Q515 616 482 625T410 635Q374 635 349 624T312 594T295 561T290 532Q290 505 303 482T342 442T378 419T409 404Q435 391 451 383T494 357T535 323T562 282T574 231Q574 133 464 56T220 -22Q138 -22 78 21T18 123Q18 184 61 227T156 274Q178 274 178 263Q178 260 177 258Q172 247 164 239T151 227T136 218L127 213L124 202Q118 186 118 163Q120 124 165 86T292 48Q374 48 423 86T473 186V193Q473 267 347 327Q268 364 239 389Q191 431 191 486Q191 547 242 600T356 679T470 705Q472 705 478 705T489 704Q551 704 596 682T642 610Q642 566 621 545Q592 516 554 512Z"></path><path stroke-width="0" id="E287-MJCAL-41" d="M576 668Q576 688 606 708T660 728Q676 728 675 712V571Q675 409 688 252Q696 122 720 57Q722 53 723 50T728 46T732 43T737 41T743 39L754 45Q788 61 803 61Q819 61 819 47Q818 43 814 35Q799 15 755 -7T675 -30Q659 -30 648 -25T630 -8T621 11T614 34Q603 77 599 106T594 146T591 160V163H460L329 164L316 145Q241 35 196 -7T119 -50T59 -24T30 43Q30 75 46 100T74 125Q81 125 83 120T88 104T96 84Q118 57 151 57Q189 57 277 182Q432 400 542 625L559 659H567Q574 659 575 660T576 668ZM584 249Q579 333 577 386T575 473T574 520V581L563 560Q497 426 412 290L372 228L370 224H371L383 228L393 232H586L584 249Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><g transform="translate(40543,0)"><g id="mjx-eqn-13" transform="translate(0,-25)"><use xlink:href="#E287-MJMAIN-28"></use><use xlink:href="#E287-MJMAIN-31" x="389" y="0"></use><use xlink:href="#E287-MJMAIN-33" x="889" y="0"></use><use xlink:href="#E287-MJMAIN-29" x="1389" y="0"></use></g></g><g transform="translate(12442,0)"><g transform="translate(-19,0)"><g transform="translate(0,-25)"><use xlink:href="#E287-MJMATHI-61" x="0" y="0"></use><use xlink:href="#E287-MJMAIN-28" x="529" y="0"></use><use xlink:href="#E287-MJMATHI-73" x="918" y="0"></use><use xlink:href="#E287-MJMAIN-2C" x="1387" y="0"></use><use xlink:href="#E287-MJMATHI-61" x="1831" y="0"></use><use xlink:href="#E287-MJMAIN-29" x="2360" y="0"></use><use xlink:href="#E287-MJMAIN-3D" x="3027" y="0"></use><use xlink:href="#E287-MJMATHI-71" x="4083" y="0"></use><use xlink:href="#E287-MJMAIN-28" x="4543" y="0"></use><use xlink:href="#E287-MJMATHI-73" x="4932" y="0"></use><use xlink:href="#E287-MJMAIN-2C" x="5401" y="0"></use><use xlink:href="#E287-MJMATHI-61" x="5845" y="0"></use><use xlink:href="#E287-MJMAIN-29" x="6374" y="0"></use><use xlink:href="#E287-MJMAIN-2212" x="6986" y="0"></use><use xlink:href="#E287-MJMATHI-76" x="7986" y="0"></use><use xlink:href="#E287-MJMAIN-28" x="8471" y="0"></use><use xlink:href="#E287-MJMATHI-73" x="8860" y="0"></use><use xlink:href="#E287-MJMAIN-29" x="9329" y="0"></use><use xlink:href="#E287-MJMAIN-2C" x="9996" y="0"></use><use xlink:href="#E287-MJMATHI-73" x="12440" y="0"></use><use xlink:href="#E287-MJMAIN-2208" x="13187" y="0"></use><use xlink:href="#E287-MJCAL-53" x="14132" y="0"></use><use xlink:href="#E287-MJMAIN-2C" x="14774" y="0"></use><use xlink:href="#E287-MJMATHI-61" x="15219" y="0"></use><use xlink:href="#E287-MJMAIN-2208" x="16025" y="0"></use><use xlink:href="#E287-MJCAL-41" x="16970" y="0"></use></g></g></g></g></svg></span></div><script type="math/tex; mode=display" id="MathJax-Element-260">a(s,a) = q(s,a) - v(s)\;, \qquad s \in \mathcal S,a \in \mathcal A</script></div></div><p><span>对偶 Q 网络仍然用 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="4.805ex" height="2.71ex" viewBox="0 -832.7 2069 1166.9" role="img" focusable="false" style="vertical-align: -0.776ex;"><defs><path stroke-width="0" id="E349-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E349-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E349-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E349-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E349-MJMATHI-71" x="0" y="0"></use><use xlink:href="#E349-MJMAIN-28" x="460" y="0"></use><use xlink:href="#E349-MJMAINB-77" x="849" y="0"></use><use xlink:href="#E349-MJMAIN-29" x="1680" y="0"></use></g></svg></span><script type="math/tex">q(\bold w)</script><span> 来估计动作价值，只不过此时其表达式为 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="31.46ex" height="2.71ex" viewBox="0 -832.7 13545.3 1166.9" role="img" focusable="false" style="vertical-align: -0.776ex;"><defs><path stroke-width="0" id="E346-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E346-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E346-MJMATHI-73" d="M131 289Q131 321 147 354T203 415T300 442Q362 442 390 415T419 355Q419 323 402 308T364 292Q351 292 340 300T328 326Q328 342 337 354T354 372T367 378Q368 378 368 379Q368 382 361 388T336 399T297 405Q249 405 227 379T204 326Q204 301 223 291T278 274T330 259Q396 230 396 163Q396 135 385 107T352 51T289 7T195 -10Q118 -10 86 19T53 87Q53 126 74 143T118 160Q133 160 146 151T160 120Q160 94 142 76T111 58Q109 57 108 57T107 55Q108 52 115 47T146 34T201 27Q237 27 263 38T301 66T318 97T323 122Q323 150 302 164T254 181T195 196T148 231Q131 256 131 289Z"></path><path stroke-width="0" id="E346-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E346-MJMATHI-61" d="M33 157Q33 258 109 349T280 441Q331 441 370 392Q386 422 416 422Q429 422 439 414T449 394Q449 381 412 234T374 68Q374 43 381 35T402 26Q411 27 422 35Q443 55 463 131Q469 151 473 152Q475 153 483 153H487Q506 153 506 144Q506 138 501 117T481 63T449 13Q436 0 417 -8Q409 -10 393 -10Q359 -10 336 5T306 36L300 51Q299 52 296 50Q294 48 292 46Q233 -10 172 -10Q117 -10 75 30T33 157ZM351 328Q351 334 346 350T323 385T277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q217 26 254 59T298 110Q300 114 325 217T351 328Z"></path><path stroke-width="0" id="E346-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E346-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E346-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E346-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E346-MJMATHI-76" d="M173 380Q173 405 154 405Q130 405 104 376T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Q21 294 29 316T53 368T97 419T160 441Q202 441 225 417T249 361Q249 344 246 335Q246 329 231 291T200 202T182 113Q182 86 187 69Q200 26 250 26Q287 26 319 60T369 139T398 222T409 277Q409 300 401 317T383 343T365 361T357 383Q357 405 376 424T417 443Q436 443 451 425T467 367Q467 340 455 284T418 159T347 40T241 -11Q177 -11 139 22Q102 54 102 117Q102 148 110 181T151 298Q173 362 173 380Z"></path><path stroke-width="0" id="E346-MJMAIN-2B" d="M56 237T56 250T70 270H369V420L370 570Q380 583 389 583Q402 583 409 568V270H707Q722 262 722 250T707 230H409V-68Q401 -82 391 -82H389H387Q375 -82 369 -68V230H70Q56 237 56 250Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E346-MJMATHI-71" x="0" y="0"></use><use xlink:href="#E346-MJMAIN-28" x="460" y="0"></use><use xlink:href="#E346-MJMATHI-73" x="849" y="0"></use><use xlink:href="#E346-MJMAIN-2C" x="1318" y="0"></use><use xlink:href="#E346-MJMATHI-61" x="1762" y="0"></use><use xlink:href="#E346-MJMAIN-3B" x="2291" y="0"></use><use xlink:href="#E346-MJMAINB-77" x="2736" y="0"></use><use xlink:href="#E346-MJMAIN-29" x="3567" y="0"></use><use xlink:href="#E346-MJMAIN-3D" x="4234" y="0"></use><use xlink:href="#E346-MJMATHI-76" x="5289" y="0"></use><use xlink:href="#E346-MJMAIN-28" x="5774" y="0"></use><use xlink:href="#E346-MJMATHI-73" x="6163" y="0"></use><use xlink:href="#E346-MJMAIN-3B" x="6632" y="0"></use><use xlink:href="#E346-MJMAINB-77" x="7077" y="0"></use><use xlink:href="#E346-MJMAIN-29" x="7908" y="0"></use><use xlink:href="#E346-MJMAIN-2B" x="8519" y="0"></use><use xlink:href="#E346-MJMATHI-61" x="9520" y="0"></use><use xlink:href="#E346-MJMAIN-28" x="10049" y="0"></use><use xlink:href="#E346-MJMATHI-73" x="10438" y="0"></use><use xlink:href="#E346-MJMAIN-2C" x="10907" y="0"></use><use xlink:href="#E346-MJMATHI-61" x="11351" y="0"></use><use xlink:href="#E346-MJMAIN-3B" x="11880" y="0"></use><use xlink:href="#E346-MJMAINB-77" x="12325" y="0"></use><use xlink:href="#E346-MJMAIN-29" x="13156" y="0"></use></g></svg></span><script type="math/tex">q(s,a;\bold w) = v(s; \bold w) + a(s,a;\bold w)</script><span> ，训练过程中 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="4.864ex" height="2.71ex" viewBox="0 -832.7 2094 1166.9" role="img" focusable="false" style="vertical-align: -0.776ex;"><defs><path stroke-width="0" id="E350-MJMATHI-76" d="M173 380Q173 405 154 405Q130 405 104 376T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Q21 294 29 316T53 368T97 419T160 441Q202 441 225 417T249 361Q249 344 246 335Q246 329 231 291T200 202T182 113Q182 86 187 69Q200 26 250 26Q287 26 319 60T369 139T398 222T409 277Q409 300 401 317T383 343T365 361T357 383Q357 405 376 424T417 443Q436 443 451 425T467 367Q467 340 455 284T418 159T347 40T241 -11Q177 -11 139 22Q102 54 102 117Q102 148 110 181T151 298Q173 362 173 380Z"></path><path stroke-width="0" id="E350-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E350-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E350-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E350-MJMATHI-76" x="0" y="0"></use><use xlink:href="#E350-MJMAIN-28" x="485" y="0"></use><use xlink:href="#E350-MJMAINB-77" x="874" y="0"></use><use xlink:href="#E350-MJMAIN-29" x="1705" y="0"></use></g></svg></span><script type="math/tex">v(\bold w)</script><span> 和 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="4.966ex" height="2.71ex" viewBox="0 -832.7 2138 1166.9" role="img" focusable="false" style="vertical-align: -0.776ex;"><defs><path stroke-width="0" id="E351-MJMATHI-61" d="M33 157Q33 258 109 349T280 441Q331 441 370 392Q386 422 416 422Q429 422 439 414T449 394Q449 381 412 234T374 68Q374 43 381 35T402 26Q411 27 422 35Q443 55 463 131Q469 151 473 152Q475 153 483 153H487Q506 153 506 144Q506 138 501 117T481 63T449 13Q436 0 417 -8Q409 -10 393 -10Q359 -10 336 5T306 36L300 51Q299 52 296 50Q294 48 292 46Q233 -10 172 -10Q117 -10 75 30T33 157ZM351 328Q351 334 346 350T323 385T277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q217 26 254 59T298 110Q300 114 325 217T351 328Z"></path><path stroke-width="0" id="E351-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E351-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E351-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E351-MJMATHI-61" x="0" y="0"></use><use xlink:href="#E351-MJMAIN-28" x="529" y="0"></use><use xlink:href="#E351-MJMAINB-77" x="918" y="0"></use><use xlink:href="#E351-MJMAIN-29" x="1749" y="0"></use></g></svg></span><script type="math/tex">a(\bold w)</script><span> 是共同训练的，和单独训练普通深度 Q 网络并无不同之处。</span></p><p><span>由于同一个 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="4.805ex" height="2.71ex" viewBox="0 -832.7 2069 1166.9" role="img" focusable="false" style="vertical-align: -0.776ex;"><defs><path stroke-width="0" id="E349-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E349-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E349-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E349-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E349-MJMATHI-71" x="0" y="0"></use><use xlink:href="#E349-MJMAIN-28" x="460" y="0"></use><use xlink:href="#E349-MJMAINB-77" x="849" y="0"></use><use xlink:href="#E349-MJMAIN-29" x="1680" y="0"></use></g></svg></span><script type="math/tex">q(\bold w)</script><span> 存在着无穷多种分解为 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="4.864ex" height="2.71ex" viewBox="0 -832.7 2094 1166.9" role="img" focusable="false" style="vertical-align: -0.776ex;"><defs><path stroke-width="0" id="E350-MJMATHI-76" d="M173 380Q173 405 154 405Q130 405 104 376T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Q21 294 29 316T53 368T97 419T160 441Q202 441 225 417T249 361Q249 344 246 335Q246 329 231 291T200 202T182 113Q182 86 187 69Q200 26 250 26Q287 26 319 60T369 139T398 222T409 277Q409 300 401 317T383 343T365 361T357 383Q357 405 376 424T417 443Q436 443 451 425T467 367Q467 340 455 284T418 159T347 40T241 -11Q177 -11 139 22Q102 54 102 117Q102 148 110 181T151 298Q173 362 173 380Z"></path><path stroke-width="0" id="E350-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E350-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E350-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E350-MJMATHI-76" x="0" y="0"></use><use xlink:href="#E350-MJMAIN-28" x="485" y="0"></use><use xlink:href="#E350-MJMAINB-77" x="874" y="0"></use><use xlink:href="#E350-MJMAIN-29" x="1705" y="0"></use></g></svg></span><script type="math/tex">v(\bold w)</script><span> 和 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="4.966ex" height="2.71ex" viewBox="0 -832.7 2138 1166.9" role="img" focusable="false" style="vertical-align: -0.776ex;"><defs><path stroke-width="0" id="E351-MJMATHI-61" d="M33 157Q33 258 109 349T280 441Q331 441 370 392Q386 422 416 422Q429 422 439 414T449 394Q449 381 412 234T374 68Q374 43 381 35T402 26Q411 27 422 35Q443 55 463 131Q469 151 473 152Q475 153 483 153H487Q506 153 506 144Q506 138 501 117T481 63T449 13Q436 0 417 -8Q409 -10 393 -10Q359 -10 336 5T306 36L300 51Q299 52 296 50Q294 48 292 46Q233 -10 172 -10Q117 -10 75 30T33 157ZM351 328Q351 334 346 350T323 385T277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q217 26 254 59T298 110Q300 114 325 217T351 328Z"></path><path stroke-width="0" id="E351-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E351-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E351-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E351-MJMATHI-61" x="0" y="0"></use><use xlink:href="#E351-MJMAIN-28" x="529" y="0"></use><use xlink:href="#E351-MJMAINB-77" x="918" y="0"></use><use xlink:href="#E351-MJMAIN-29" x="1749" y="0"></use></g></svg></span><script type="math/tex">a(\bold w)</script><span> 的方式，那么可以通过增加一个由优势函数导出的量，使得等效的优势函数满足固定的特征，使得分解唯一。常见的方法由以下两种：</span></p><ul><li><p><span>考虑优势函数的最大值，令</span></p><div contenteditable="false" spellcheck="false" class="mathjax-block md-end-block md-math-block md-rawblock" id="mathjax-n108" cid="n108" mdtype="math_block"><div class="md-rawblock-container md-math-container" tabindex="-1"><div class="MathJax_SVG_Display"><span class="MathJax_SVG" id="MathJax-Element-261-Frame" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="94.63ex" height="4.061ex" viewBox="0 -1123.4 40743.4 1748.4" role="img" focusable="false" style="vertical-align: -1.452ex; max-width: 100%;"><defs><path stroke-width="0" id="E288-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E288-MJMAIN-31" d="M213 578L200 573Q186 568 160 563T102 556H83V602H102Q149 604 189 617T245 641T273 663Q275 666 285 666Q294 666 302 660V361L303 61Q310 54 315 52T339 48T401 46H427V0H416Q395 3 257 3Q121 3 100 0H88V46H114Q136 46 152 46T177 47T193 50T201 52T207 57T213 61V578Z"></path><path stroke-width="0" id="E288-MJMAIN-34" d="M462 0Q444 3 333 3Q217 3 199 0H190V46H221Q241 46 248 46T265 48T279 53T286 61Q287 63 287 115V165H28V211L179 442Q332 674 334 675Q336 677 355 677H373L379 671V211H471V165H379V114Q379 73 379 66T385 54Q393 47 442 46H471V0H462ZM293 211V545L74 212L183 211H293Z"></path><path stroke-width="0" id="E288-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E288-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E288-MJMATHI-73" d="M131 289Q131 321 147 354T203 415T300 442Q362 442 390 415T419 355Q419 323 402 308T364 292Q351 292 340 300T328 326Q328 342 337 354T354 372T367 378Q368 378 368 379Q368 382 361 388T336 399T297 405Q249 405 227 379T204 326Q204 301 223 291T278 274T330 259Q396 230 396 163Q396 135 385 107T352 51T289 7T195 -10Q118 -10 86 19T53 87Q53 126 74 143T118 160Q133 160 146 151T160 120Q160 94 142 76T111 58Q109 57 108 57T107 55Q108 52 115 47T146 34T201 27Q237 27 263 38T301 66T318 97T323 122Q323 150 302 164T254 181T195 196T148 231Q131 256 131 289Z"></path><path stroke-width="0" id="E288-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E288-MJMATHI-61" d="M33 157Q33 258 109 349T280 441Q331 441 370 392Q386 422 416 422Q429 422 439 414T449 394Q449 381 412 234T374 68Q374 43 381 35T402 26Q411 27 422 35Q443 55 463 131Q469 151 473 152Q475 153 483 153H487Q506 153 506 144Q506 138 501 117T481 63T449 13Q436 0 417 -8Q409 -10 393 -10Q359 -10 336 5T306 36L300 51Q299 52 296 50Q294 48 292 46Q233 -10 172 -10Q117 -10 75 30T33 157ZM351 328Q351 334 346 350T323 385T277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q217 26 254 59T298 110Q300 114 325 217T351 328Z"></path><path stroke-width="0" id="E288-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E288-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E288-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E288-MJMATHI-76" d="M173 380Q173 405 154 405Q130 405 104 376T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Q21 294 29 316T53 368T97 419T160 441Q202 441 225 417T249 361Q249 344 246 335Q246 329 231 291T200 202T182 113Q182 86 187 69Q200 26 250 26Q287 26 319 60T369 139T398 222T409 277Q409 300 401 317T383 343T365 361T357 383Q357 405 376 424T417 443Q436 443 451 425T467 367Q467 340 455 284T418 159T347 40T241 -11Q177 -11 139 22Q102 54 102 117Q102 148 110 181T151 298Q173 362 173 380Z"></path><path stroke-width="0" id="E288-MJMAIN-2B" d="M56 237T56 250T70 270H369V420L370 570Q380 583 389 583Q402 583 409 568V270H707Q722 262 722 250T707 230H409V-68Q401 -82 391 -82H389H387Q375 -82 369 -68V230H70Q56 237 56 250Z"></path><path stroke-width="0" id="E288-MJMAIN-2212" d="M84 237T84 250T98 270H679Q694 262 694 250T679 230H98Q84 237 84 250Z"></path><path stroke-width="0" id="E288-MJMAIN-6D" d="M41 46H55Q94 46 102 60V68Q102 77 102 91T102 122T103 161T103 203Q103 234 103 269T102 328V351Q99 370 88 376T43 385H25V408Q25 431 27 431L37 432Q47 433 65 434T102 436Q119 437 138 438T167 441T178 442H181V402Q181 364 182 364T187 369T199 384T218 402T247 421T285 437Q305 442 336 442Q351 442 364 440T387 434T406 426T421 417T432 406T441 395T448 384T452 374T455 366L457 361L460 365Q463 369 466 373T475 384T488 397T503 410T523 422T546 432T572 439T603 442Q729 442 740 329Q741 322 741 190V104Q741 66 743 59T754 49Q775 46 803 46H819V0H811L788 1Q764 2 737 2T699 3Q596 3 587 0H579V46H595Q656 46 656 62Q657 64 657 200Q656 335 655 343Q649 371 635 385T611 402T585 404Q540 404 506 370Q479 343 472 315T464 232V168V108Q464 78 465 68T468 55T477 49Q498 46 526 46H542V0H534L510 1Q487 2 460 2T422 3Q319 3 310 0H302V46H318Q379 46 379 62Q380 64 380 200Q379 335 378 343Q372 371 358 385T334 402T308 404Q263 404 229 370Q202 343 195 315T187 232V168V108Q187 78 188 68T191 55T200 49Q221 46 249 46H265V0H257L234 1Q210 2 183 2T145 3Q42 3 33 0H25V46H41Z"></path><path stroke-width="0" id="E288-MJMAIN-61" d="M137 305T115 305T78 320T63 359Q63 394 97 421T218 448Q291 448 336 416T396 340Q401 326 401 309T402 194V124Q402 76 407 58T428 40Q443 40 448 56T453 109V145H493V106Q492 66 490 59Q481 29 455 12T400 -6T353 12T329 54V58L327 55Q325 52 322 49T314 40T302 29T287 17T269 6T247 -2T221 -8T190 -11Q130 -11 82 20T34 107Q34 128 41 147T68 188T116 225T194 253T304 268H318V290Q318 324 312 340Q290 411 215 411Q197 411 181 410T156 406T148 403Q170 388 170 359Q170 334 154 320ZM126 106Q126 75 150 51T209 26Q247 26 276 49T315 109Q317 116 318 175Q318 233 317 233Q309 233 296 232T251 223T193 203T147 166T126 106Z"></path><path stroke-width="0" id="E288-MJMAIN-78" d="M201 0Q189 3 102 3Q26 3 17 0H11V46H25Q48 47 67 52T96 61T121 78T139 96T160 122T180 150L226 210L168 288Q159 301 149 315T133 336T122 351T113 363T107 370T100 376T94 379T88 381T80 383Q74 383 44 385H16V431H23Q59 429 126 429Q219 429 229 431H237V385Q201 381 201 369Q201 367 211 353T239 315T268 274L272 270L297 304Q329 345 329 358Q329 364 327 369T322 376T317 380T310 384L307 385H302V431H309Q324 428 408 428Q487 428 493 431H499V385H492Q443 385 411 368Q394 360 377 341T312 257L296 236L358 151Q424 61 429 57T446 50Q464 46 499 46H516V0H510H502Q494 1 482 1T457 2T432 2T414 3Q403 3 377 3T327 1L304 0H295V46H298Q309 46 320 51T331 63Q331 65 291 120L250 175Q249 174 219 133T185 88Q181 83 181 74Q181 63 188 55T206 46Q208 46 208 23V0H201Z"></path><path stroke-width="0" id="E288-MJMAIN-2208" d="M84 250Q84 372 166 450T360 539Q361 539 377 539T419 540T469 540H568Q583 532 583 520Q583 511 570 501L466 500Q355 499 329 494Q280 482 242 458T183 409T147 354T129 306T124 272V270H568Q583 262 583 250T568 230H124V228Q124 207 134 177T167 112T231 48T328 7Q355 1 466 0H570Q583 -10 583 -20Q583 -32 568 -40H471Q464 -40 446 -40T417 -41Q262 -41 172 45Q84 127 84 250Z"></path><path stroke-width="0" id="E288-MJCAL-41" d="M576 668Q576 688 606 708T660 728Q676 728 675 712V571Q675 409 688 252Q696 122 720 57Q722 53 723 50T728 46T732 43T737 41T743 39L754 45Q788 61 803 61Q819 61 819 47Q818 43 814 35Q799 15 755 -7T675 -30Q659 -30 648 -25T630 -8T621 11T614 34Q603 77 599 106T594 146T591 160V163H460L329 164L316 145Q241 35 196 -7T119 -50T59 -24T30 43Q30 75 46 100T74 125Q81 125 83 120T88 104T96 84Q118 57 151 57Q189 57 277 182Q432 400 542 625L559 659H567Q574 659 575 660T576 668ZM584 249Q579 333 577 386T575 473T574 520V581L563 560Q497 426 412 290L372 228L370 224H371L383 228L393 232H586L584 249Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><g transform="translate(38965,0)"><g id="mjx-eqn-14" transform="translate(0,264)"><use xlink:href="#E288-MJMAIN-28"></use><use xlink:href="#E288-MJMAIN-31" x="389" y="0"></use><use xlink:href="#E288-MJMAIN-34" x="889" y="0"></use><use xlink:href="#E288-MJMAIN-29" x="1389" y="0"></use></g></g><g transform="translate(10137,0)"><g transform="translate(-19,0)"><g transform="translate(0,264)"><use xlink:href="#E288-MJMATHI-71" x="0" y="0"></use><use xlink:href="#E288-MJMAIN-28" x="460" y="0"></use><use xlink:href="#E288-MJMATHI-73" x="849" y="0"></use><use xlink:href="#E288-MJMAIN-2C" x="1318" y="0"></use><use xlink:href="#E288-MJMATHI-61" x="1762" y="0"></use><use xlink:href="#E288-MJMAIN-3B" x="2291" y="0"></use><use xlink:href="#E288-MJMAINB-77" x="2736" y="0"></use><use xlink:href="#E288-MJMAIN-29" x="3567" y="0"></use><use xlink:href="#E288-MJMAIN-3D" x="4234" y="0"></use><use xlink:href="#E288-MJMATHI-76" x="5289" y="0"></use><use xlink:href="#E288-MJMAIN-28" x="5774" y="0"></use><use xlink:href="#E288-MJMATHI-73" x="6163" y="0"></use><use xlink:href="#E288-MJMAIN-3B" x="6632" y="0"></use><use xlink:href="#E288-MJMAINB-77" x="7077" y="0"></use><use xlink:href="#E288-MJMAIN-29" x="7908" y="0"></use><use xlink:href="#E288-MJMAIN-2B" x="8519" y="0"></use><use xlink:href="#E288-MJMATHI-61" x="9520" y="0"></use><use xlink:href="#E288-MJMAIN-28" x="10049" y="0"></use><use xlink:href="#E288-MJMATHI-73" x="10438" y="0"></use><use xlink:href="#E288-MJMAIN-2C" x="10907" y="0"></use><use xlink:href="#E288-MJMATHI-61" x="11351" y="0"></use><use xlink:href="#E288-MJMAIN-3B" x="11880" y="0"></use><use xlink:href="#E288-MJMAINB-77" x="12325" y="0"></use><use xlink:href="#E288-MJMAIN-29" x="13156" y="0"></use><use xlink:href="#E288-MJMAIN-2212" x="13767" y="0"></use><g transform="translate(14767,0)"><use xlink:href="#E288-MJMAIN-6D"></use><use xlink:href="#E288-MJMAIN-61" x="833" y="0"></use><use xlink:href="#E288-MJMAIN-78" x="1333" y="0"></use><g transform="translate(218,-693)"><use transform="scale(0.707)" xlink:href="#E288-MJMATHI-61" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E288-MJMAIN-2208" x="529" y="0"></use><use transform="scale(0.707)" xlink:href="#E288-MJCAL-41" x="1196" y="0"></use></g></g><use xlink:href="#E288-MJMATHI-61" x="16795" y="0"></use><use xlink:href="#E288-MJMAIN-28" x="17324" y="0"></use><use xlink:href="#E288-MJMATHI-73" x="17713" y="0"></use><use xlink:href="#E288-MJMAIN-2C" x="18182" y="0"></use><use xlink:href="#E288-MJMATHI-61" x="18627" y="0"></use><use xlink:href="#E288-MJMAIN-3B" x="19156" y="0"></use><use xlink:href="#E288-MJMAINB-77" x="19600" y="0"></use><use xlink:href="#E288-MJMAIN-29" x="20431" y="0"></use></g></g></g></g></svg></span></div><script type="math/tex; mode=display" id="MathJax-Element-261">q(s,a;\bold w) = v(s;\bold w) + a(s,a;\bold w) - \max_{a \in \mathcal A} a(s,a;\bold w)</script></div></div><p><span>使得等效优势函数 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="41.654ex" height="3.964ex" viewBox="0 -832.7 17934.4 1706.8" role="img" focusable="false" style="vertical-align: -2.03ex;"><defs><path stroke-width="0" id="E352-MJMATHI-61" d="M33 157Q33 258 109 349T280 441Q331 441 370 392Q386 422 416 422Q429 422 439 414T449 394Q449 381 412 234T374 68Q374 43 381 35T402 26Q411 27 422 35Q443 55 463 131Q469 151 473 152Q475 153 483 153H487Q506 153 506 144Q506 138 501 117T481 63T449 13Q436 0 417 -8Q409 -10 393 -10Q359 -10 336 5T306 36L300 51Q299 52 296 50Q294 48 292 46Q233 -10 172 -10Q117 -10 75 30T33 157ZM351 328Q351 334 346 350T323 385T277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q217 26 254 59T298 110Q300 114 325 217T351 328Z"></path><path stroke-width="0" id="E352-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E352-MJMATHI-73" d="M131 289Q131 321 147 354T203 415T300 442Q362 442 390 415T419 355Q419 323 402 308T364 292Q351 292 340 300T328 326Q328 342 337 354T354 372T367 378Q368 378 368 379Q368 382 361 388T336 399T297 405Q249 405 227 379T204 326Q204 301 223 291T278 274T330 259Q396 230 396 163Q396 135 385 107T352 51T289 7T195 -10Q118 -10 86 19T53 87Q53 126 74 143T118 160Q133 160 146 151T160 120Q160 94 142 76T111 58Q109 57 108 57T107 55Q108 52 115 47T146 34T201 27Q237 27 263 38T301 66T318 97T323 122Q323 150 302 164T254 181T195 196T148 231Q131 256 131 289Z"></path><path stroke-width="0" id="E352-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E352-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E352-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E352-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E352-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E352-MJMAIN-2212" d="M84 237T84 250T98 270H679Q694 262 694 250T679 230H98Q84 237 84 250Z"></path><path stroke-width="0" id="E352-MJMAIN-6D" d="M41 46H55Q94 46 102 60V68Q102 77 102 91T102 122T103 161T103 203Q103 234 103 269T102 328V351Q99 370 88 376T43 385H25V408Q25 431 27 431L37 432Q47 433 65 434T102 436Q119 437 138 438T167 441T178 442H181V402Q181 364 182 364T187 369T199 384T218 402T247 421T285 437Q305 442 336 442Q351 442 364 440T387 434T406 426T421 417T432 406T441 395T448 384T452 374T455 366L457 361L460 365Q463 369 466 373T475 384T488 397T503 410T523 422T546 432T572 439T603 442Q729 442 740 329Q741 322 741 190V104Q741 66 743 59T754 49Q775 46 803 46H819V0H811L788 1Q764 2 737 2T699 3Q596 3 587 0H579V46H595Q656 46 656 62Q657 64 657 200Q656 335 655 343Q649 371 635 385T611 402T585 404Q540 404 506 370Q479 343 472 315T464 232V168V108Q464 78 465 68T468 55T477 49Q498 46 526 46H542V0H534L510 1Q487 2 460 2T422 3Q319 3 310 0H302V46H318Q379 46 379 62Q380 64 380 200Q379 335 378 343Q372 371 358 385T334 402T308 404Q263 404 229 370Q202 343 195 315T187 232V168V108Q187 78 188 68T191 55T200 49Q221 46 249 46H265V0H257L234 1Q210 2 183 2T145 3Q42 3 33 0H25V46H41Z"></path><path stroke-width="0" id="E352-MJMAIN-61" d="M137 305T115 305T78 320T63 359Q63 394 97 421T218 448Q291 448 336 416T396 340Q401 326 401 309T402 194V124Q402 76 407 58T428 40Q443 40 448 56T453 109V145H493V106Q492 66 490 59Q481 29 455 12T400 -6T353 12T329 54V58L327 55Q325 52 322 49T314 40T302 29T287 17T269 6T247 -2T221 -8T190 -11Q130 -11 82 20T34 107Q34 128 41 147T68 188T116 225T194 253T304 268H318V290Q318 324 312 340Q290 411 215 411Q197 411 181 410T156 406T148 403Q170 388 170 359Q170 334 154 320ZM126 106Q126 75 150 51T209 26Q247 26 276 49T315 109Q317 116 318 175Q318 233 317 233Q309 233 296 232T251 223T193 203T147 166T126 106Z"></path><path stroke-width="0" id="E352-MJMAIN-78" d="M201 0Q189 3 102 3Q26 3 17 0H11V46H25Q48 47 67 52T96 61T121 78T139 96T160 122T180 150L226 210L168 288Q159 301 149 315T133 336T122 351T113 363T107 370T100 376T94 379T88 381T80 383Q74 383 44 385H16V431H23Q59 429 126 429Q219 429 229 431H237V385Q201 381 201 369Q201 367 211 353T239 315T268 274L272 270L297 304Q329 345 329 358Q329 364 327 369T322 376T317 380T310 384L307 385H302V431H309Q324 428 408 428Q487 428 493 431H499V385H492Q443 385 411 368Q394 360 377 341T312 257L296 236L358 151Q424 61 429 57T446 50Q464 46 499 46H516V0H510H502Q494 1 482 1T457 2T432 2T414 3Q403 3 377 3T327 1L304 0H295V46H298Q309 46 320 51T331 63Q331 65 291 120L250 175Q249 174 219 133T185 88Q181 83 181 74Q181 63 188 55T206 46Q208 46 208 23V0H201Z"></path><path stroke-width="0" id="E352-MJMAIN-2208" d="M84 250Q84 372 166 450T360 539Q361 539 377 539T419 540T469 540H568Q583 532 583 520Q583 511 570 501L466 500Q355 499 329 494Q280 482 242 458T183 409T147 354T129 306T124 272V270H568Q583 262 583 250T568 230H124V228Q124 207 134 177T167 112T231 48T328 7Q355 1 466 0H570Q583 -10 583 -20Q583 -32 568 -40H471Q464 -40 446 -40T417 -41Q262 -41 172 45Q84 127 84 250Z"></path><path stroke-width="0" id="E352-MJCAL-41" d="M576 668Q576 688 606 708T660 728Q676 728 675 712V571Q675 409 688 252Q696 122 720 57Q722 53 723 50T728 46T732 43T737 41T743 39L754 45Q788 61 803 61Q819 61 819 47Q818 43 814 35Q799 15 755 -7T675 -30Q659 -30 648 -25T630 -8T621 11T614 34Q603 77 599 106T594 146T591 160V163H460L329 164L316 145Q241 35 196 -7T119 -50T59 -24T30 43Q30 75 46 100T74 125Q81 125 83 120T88 104T96 84Q118 57 151 57Q189 57 277 182Q432 400 542 625L559 659H567Q574 659 575 660T576 668ZM584 249Q579 333 577 386T575 473T574 520V581L563 560Q497 426 412 290L372 228L370 224H371L383 228L393 232H586L584 249Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E352-MJMATHI-61" x="0" y="0"></use><g transform="translate(529,-150)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(29.368) matrix(1 0 0 -1 0 0)">等</text><g transform="translate(587,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(29.368) matrix(1 0 0 -1 0 0)">效</text></g></g><use xlink:href="#E352-MJMAIN-28" x="1803" y="0"></use><use xlink:href="#E352-MJMATHI-73" x="2192" y="0"></use><use xlink:href="#E352-MJMAIN-2C" x="2661" y="0"></use><use xlink:href="#E352-MJMATHI-61" x="3106" y="0"></use><use xlink:href="#E352-MJMAIN-3B" x="3635" y="0"></use><use xlink:href="#E352-MJMAINB-77" x="4080" y="0"></use><use xlink:href="#E352-MJMAIN-29" x="4911" y="0"></use><use xlink:href="#E352-MJMAIN-3D" x="5577" y="0"></use><use xlink:href="#E352-MJMATHI-61" x="6633" y="0"></use><use xlink:href="#E352-MJMAIN-28" x="7162" y="0"></use><use xlink:href="#E352-MJMATHI-73" x="7551" y="0"></use><use xlink:href="#E352-MJMAIN-2C" x="8020" y="0"></use><use xlink:href="#E352-MJMATHI-61" x="8465" y="0"></use><use xlink:href="#E352-MJMAIN-3B" x="8994" y="0"></use><use xlink:href="#E352-MJMAINB-77" x="9438" y="0"></use><use xlink:href="#E352-MJMAIN-29" x="10269" y="0"></use><use xlink:href="#E352-MJMAIN-2212" x="10881" y="0"></use><g transform="translate(11881,0)"><use xlink:href="#E352-MJMAIN-6D"></use><use xlink:href="#E352-MJMAIN-61" x="833" y="0"></use><use xlink:href="#E352-MJMAIN-78" x="1333" y="0"></use><g transform="translate(218,-693)"><use transform="scale(0.707)" xlink:href="#E352-MJMATHI-61" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E352-MJMAIN-2208" x="529" y="0"></use><use transform="scale(0.707)" xlink:href="#E352-MJCAL-41" x="1196" y="0"></use></g></g><use xlink:href="#E352-MJMATHI-61" x="13909" y="0"></use><use xlink:href="#E352-MJMAIN-28" x="14438" y="0"></use><use xlink:href="#E352-MJMATHI-73" x="14827" y="0"></use><use xlink:href="#E352-MJMAIN-2C" x="15296" y="0"></use><use xlink:href="#E352-MJMATHI-61" x="15740" y="0"></use><use xlink:href="#E352-MJMAIN-3B" x="16269" y="0"></use><use xlink:href="#E352-MJMAINB-77" x="16714" y="0"></use><use xlink:href="#E352-MJMAIN-29" x="17545" y="0"></use></g></svg></span><script type="math/tex">\displaystyle a_{等效}(s,a;\bold w) = a(s,a;\bold w) - \max_{a\in\mathcal A}a(s,a;\bold w)</script><span> 满足 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="28.376ex" height="3.964ex" viewBox="0 -832.7 12217.3 1706.8" role="img" focusable="false" style="vertical-align: -2.03ex;"><defs><path stroke-width="0" id="E353-MJMAIN-6D" d="M41 46H55Q94 46 102 60V68Q102 77 102 91T102 122T103 161T103 203Q103 234 103 269T102 328V351Q99 370 88 376T43 385H25V408Q25 431 27 431L37 432Q47 433 65 434T102 436Q119 437 138 438T167 441T178 442H181V402Q181 364 182 364T187 369T199 384T218 402T247 421T285 437Q305 442 336 442Q351 442 364 440T387 434T406 426T421 417T432 406T441 395T448 384T452 374T455 366L457 361L460 365Q463 369 466 373T475 384T488 397T503 410T523 422T546 432T572 439T603 442Q729 442 740 329Q741 322 741 190V104Q741 66 743 59T754 49Q775 46 803 46H819V0H811L788 1Q764 2 737 2T699 3Q596 3 587 0H579V46H595Q656 46 656 62Q657 64 657 200Q656 335 655 343Q649 371 635 385T611 402T585 404Q540 404 506 370Q479 343 472 315T464 232V168V108Q464 78 465 68T468 55T477 49Q498 46 526 46H542V0H534L510 1Q487 2 460 2T422 3Q319 3 310 0H302V46H318Q379 46 379 62Q380 64 380 200Q379 335 378 343Q372 371 358 385T334 402T308 404Q263 404 229 370Q202 343 195 315T187 232V168V108Q187 78 188 68T191 55T200 49Q221 46 249 46H265V0H257L234 1Q210 2 183 2T145 3Q42 3 33 0H25V46H41Z"></path><path stroke-width="0" id="E353-MJMAIN-61" d="M137 305T115 305T78 320T63 359Q63 394 97 421T218 448Q291 448 336 416T396 340Q401 326 401 309T402 194V124Q402 76 407 58T428 40Q443 40 448 56T453 109V145H493V106Q492 66 490 59Q481 29 455 12T400 -6T353 12T329 54V58L327 55Q325 52 322 49T314 40T302 29T287 17T269 6T247 -2T221 -8T190 -11Q130 -11 82 20T34 107Q34 128 41 147T68 188T116 225T194 253T304 268H318V290Q318 324 312 340Q290 411 215 411Q197 411 181 410T156 406T148 403Q170 388 170 359Q170 334 154 320ZM126 106Q126 75 150 51T209 26Q247 26 276 49T315 109Q317 116 318 175Q318 233 317 233Q309 233 296 232T251 223T193 203T147 166T126 106Z"></path><path stroke-width="0" id="E353-MJMAIN-78" d="M201 0Q189 3 102 3Q26 3 17 0H11V46H25Q48 47 67 52T96 61T121 78T139 96T160 122T180 150L226 210L168 288Q159 301 149 315T133 336T122 351T113 363T107 370T100 376T94 379T88 381T80 383Q74 383 44 385H16V431H23Q59 429 126 429Q219 429 229 431H237V385Q201 381 201 369Q201 367 211 353T239 315T268 274L272 270L297 304Q329 345 329 358Q329 364 327 369T322 376T317 380T310 384L307 385H302V431H309Q324 428 408 428Q487 428 493 431H499V385H492Q443 385 411 368Q394 360 377 341T312 257L296 236L358 151Q424 61 429 57T446 50Q464 46 499 46H516V0H510H502Q494 1 482 1T457 2T432 2T414 3Q403 3 377 3T327 1L304 0H295V46H298Q309 46 320 51T331 63Q331 65 291 120L250 175Q249 174 219 133T185 88Q181 83 181 74Q181 63 188 55T206 46Q208 46 208 23V0H201Z"></path><path stroke-width="0" id="E353-MJMATHI-61" d="M33 157Q33 258 109 349T280 441Q331 441 370 392Q386 422 416 422Q429 422 439 414T449 394Q449 381 412 234T374 68Q374 43 381 35T402 26Q411 27 422 35Q443 55 463 131Q469 151 473 152Q475 153 483 153H487Q506 153 506 144Q506 138 501 117T481 63T449 13Q436 0 417 -8Q409 -10 393 -10Q359 -10 336 5T306 36L300 51Q299 52 296 50Q294 48 292 46Q233 -10 172 -10Q117 -10 75 30T33 157ZM351 328Q351 334 346 350T323 385T277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q217 26 254 59T298 110Q300 114 325 217T351 328Z"></path><path stroke-width="0" id="E353-MJMAIN-2208" d="M84 250Q84 372 166 450T360 539Q361 539 377 539T419 540T469 540H568Q583 532 583 520Q583 511 570 501L466 500Q355 499 329 494Q280 482 242 458T183 409T147 354T129 306T124 272V270H568Q583 262 583 250T568 230H124V228Q124 207 134 177T167 112T231 48T328 7Q355 1 466 0H570Q583 -10 583 -20Q583 -32 568 -40H471Q464 -40 446 -40T417 -41Q262 -41 172 45Q84 127 84 250Z"></path><path stroke-width="0" id="E353-MJCAL-41" d="M576 668Q576 688 606 708T660 728Q676 728 675 712V571Q675 409 688 252Q696 122 720 57Q722 53 723 50T728 46T732 43T737 41T743 39L754 45Q788 61 803 61Q819 61 819 47Q818 43 814 35Q799 15 755 -7T675 -30Q659 -30 648 -25T630 -8T621 11T614 34Q603 77 599 106T594 146T591 160V163H460L329 164L316 145Q241 35 196 -7T119 -50T59 -24T30 43Q30 75 46 100T74 125Q81 125 83 120T88 104T96 84Q118 57 151 57Q189 57 277 182Q432 400 542 625L559 659H567Q574 659 575 660T576 668ZM584 249Q579 333 577 386T575 473T574 520V581L563 560Q497 426 412 290L372 228L370 224H371L383 228L393 232H586L584 249Z"></path><path stroke-width="0" id="E353-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E353-MJMATHI-73" d="M131 289Q131 321 147 354T203 415T300 442Q362 442 390 415T419 355Q419 323 402 308T364 292Q351 292 340 300T328 326Q328 342 337 354T354 372T367 378Q368 378 368 379Q368 382 361 388T336 399T297 405Q249 405 227 379T204 326Q204 301 223 291T278 274T330 259Q396 230 396 163Q396 135 385 107T352 51T289 7T195 -10Q118 -10 86 19T53 87Q53 126 74 143T118 160Q133 160 146 151T160 120Q160 94 142 76T111 58Q109 57 108 57T107 55Q108 52 115 47T146 34T201 27Q237 27 263 38T301 66T318 97T323 122Q323 150 302 164T254 181T195 196T148 231Q131 256 131 289Z"></path><path stroke-width="0" id="E353-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E353-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E353-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E353-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E353-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E353-MJMAIN-30" d="M96 585Q152 666 249 666Q297 666 345 640T423 548Q460 465 460 320Q460 165 417 83Q397 41 362 16T301 -15T250 -22Q224 -22 198 -16T137 16T82 83Q39 165 39 320Q39 494 96 585ZM321 597Q291 629 250 629Q208 629 178 597Q153 571 145 525T137 333Q137 175 145 125T181 46Q209 16 250 16Q290 16 318 46Q347 76 354 130T362 333Q362 478 354 524T321 597Z"></path><path stroke-width="0" id="E353-MJCAL-53" d="M554 512Q536 512 536 522Q536 525 539 539T542 564Q542 588 528 604Q515 616 482 625T410 635Q374 635 349 624T312 594T295 561T290 532Q290 505 303 482T342 442T378 419T409 404Q435 391 451 383T494 357T535 323T562 282T574 231Q574 133 464 56T220 -22Q138 -22 78 21T18 123Q18 184 61 227T156 274Q178 274 178 263Q178 260 177 258Q172 247 164 239T151 227T136 218L127 213L124 202Q118 186 118 163Q120 124 165 86T292 48Q374 48 423 86T473 186V193Q473 267 347 327Q268 364 239 389Q191 431 191 486Q191 547 242 600T356 679T470 705Q472 705 478 705T489 704Q551 704 596 682T642 610Q642 566 621 545Q592 516 554 512Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E353-MJMAIN-6D"></use><use xlink:href="#E353-MJMAIN-61" x="833" y="0"></use><use xlink:href="#E353-MJMAIN-78" x="1333" y="0"></use><g transform="translate(218,-693)"><use transform="scale(0.707)" xlink:href="#E353-MJMATHI-61" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E353-MJMAIN-2208" x="529" y="0"></use><use transform="scale(0.707)" xlink:href="#E353-MJCAL-41" x="1196" y="0"></use></g><g transform="translate(2027,0)"><use xlink:href="#E353-MJMATHI-61" x="0" y="0"></use><g transform="translate(529,-150)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(29.368) matrix(1 0 0 -1 0 0)">等</text><g transform="translate(587,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(29.368) matrix(1 0 0 -1 0 0)">效</text></g></g></g><use xlink:href="#E353-MJMAIN-28" x="3831" y="0"></use><use xlink:href="#E353-MJMATHI-73" x="4220" y="0"></use><use xlink:href="#E353-MJMAIN-2C" x="4689" y="0"></use><use xlink:href="#E353-MJMATHI-61" x="5134" y="0"></use><use xlink:href="#E353-MJMAIN-3B" x="5663" y="0"></use><use xlink:href="#E353-MJMAINB-77" x="6107" y="0"></use><use xlink:href="#E353-MJMAIN-29" x="6938" y="0"></use><use xlink:href="#E353-MJMAIN-3D" x="7605" y="0"></use><use xlink:href="#E353-MJMAIN-30" x="8661" y="0"></use><use xlink:href="#E353-MJMAIN-2C" x="9161" y="0"></use><use xlink:href="#E353-MJMATHI-73" x="9883" y="0"></use><use xlink:href="#E353-MJMAIN-2208" x="10630" y="0"></use><use xlink:href="#E353-MJCAL-53" x="11575" y="0"></use></g></svg></span><script type="math/tex">\displaystyle \max_{a \in \mathcal A}a_{等效}(s,a;\bold w)=0,\;s \in \mathcal S</script><span> 。</span></p></li><li><p><span>考虑优势函数的平均值，令</span></p><div contenteditable="false" spellcheck="false" class="mathjax-block md-end-block md-math-block md-rawblock" id="mathjax-n112" cid="n112" mdtype="math_block"><div class="md-rawblock-container md-math-container" tabindex="-1"><div class="MathJax_SVG_Display"><span class="MathJax_SVG" id="MathJax-Element-262-Frame" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="94.63ex" height="6.376ex" viewBox="0 -1621.8 40743.4 2745.1" role="img" focusable="false" style="vertical-align: -2.609ex; max-width: 100%;"><defs><path stroke-width="0" id="E289-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E289-MJMAIN-31" d="M213 578L200 573Q186 568 160 563T102 556H83V602H102Q149 604 189 617T245 641T273 663Q275 666 285 666Q294 666 302 660V361L303 61Q310 54 315 52T339 48T401 46H427V0H416Q395 3 257 3Q121 3 100 0H88V46H114Q136 46 152 46T177 47T193 50T201 52T207 57T213 61V578Z"></path><path stroke-width="0" id="E289-MJMAIN-35" d="M164 157Q164 133 148 117T109 101H102Q148 22 224 22Q294 22 326 82Q345 115 345 210Q345 313 318 349Q292 382 260 382H254Q176 382 136 314Q132 307 129 306T114 304Q97 304 95 310Q93 314 93 485V614Q93 664 98 664Q100 666 102 666Q103 666 123 658T178 642T253 634Q324 634 389 662Q397 666 402 666Q410 666 410 648V635Q328 538 205 538Q174 538 149 544L139 546V374Q158 388 169 396T205 412T256 420Q337 420 393 355T449 201Q449 109 385 44T229 -22Q148 -22 99 32T50 154Q50 178 61 192T84 210T107 214Q132 214 148 197T164 157Z"></path><path stroke-width="0" id="E289-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E289-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E289-MJMATHI-73" d="M131 289Q131 321 147 354T203 415T300 442Q362 442 390 415T419 355Q419 323 402 308T364 292Q351 292 340 300T328 326Q328 342 337 354T354 372T367 378Q368 378 368 379Q368 382 361 388T336 399T297 405Q249 405 227 379T204 326Q204 301 223 291T278 274T330 259Q396 230 396 163Q396 135 385 107T352 51T289 7T195 -10Q118 -10 86 19T53 87Q53 126 74 143T118 160Q133 160 146 151T160 120Q160 94 142 76T111 58Q109 57 108 57T107 55Q108 52 115 47T146 34T201 27Q237 27 263 38T301 66T318 97T323 122Q323 150 302 164T254 181T195 196T148 231Q131 256 131 289Z"></path><path stroke-width="0" id="E289-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E289-MJMATHI-61" d="M33 157Q33 258 109 349T280 441Q331 441 370 392Q386 422 416 422Q429 422 439 414T449 394Q449 381 412 234T374 68Q374 43 381 35T402 26Q411 27 422 35Q443 55 463 131Q469 151 473 152Q475 153 483 153H487Q506 153 506 144Q506 138 501 117T481 63T449 13Q436 0 417 -8Q409 -10 393 -10Q359 -10 336 5T306 36L300 51Q299 52 296 50Q294 48 292 46Q233 -10 172 -10Q117 -10 75 30T33 157ZM351 328Q351 334 346 350T323 385T277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q217 26 254 59T298 110Q300 114 325 217T351 328Z"></path><path stroke-width="0" id="E289-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E289-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E289-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E289-MJMATHI-76" d="M173 380Q173 405 154 405Q130 405 104 376T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Q21 294 29 316T53 368T97 419T160 441Q202 441 225 417T249 361Q249 344 246 335Q246 329 231 291T200 202T182 113Q182 86 187 69Q200 26 250 26Q287 26 319 60T369 139T398 222T409 277Q409 300 401 317T383 343T365 361T357 383Q357 405 376 424T417 443Q436 443 451 425T467 367Q467 340 455 284T418 159T347 40T241 -11Q177 -11 139 22Q102 54 102 117Q102 148 110 181T151 298Q173 362 173 380Z"></path><path stroke-width="0" id="E289-MJMAIN-2B" d="M56 237T56 250T70 270H369V420L370 570Q380 583 389 583Q402 583 409 568V270H707Q722 262 722 250T707 230H409V-68Q401 -82 391 -82H389H387Q375 -82 369 -68V230H70Q56 237 56 250Z"></path><path stroke-width="0" id="E289-MJMAIN-2212" d="M84 237T84 250T98 270H679Q694 262 694 250T679 230H98Q84 237 84 250Z"></path><path stroke-width="0" id="E289-MJMAIN-7C" d="M139 -249H137Q125 -249 119 -235V251L120 737Q130 750 139 750Q152 750 159 735V-235Q151 -249 141 -249H139Z"></path><path stroke-width="0" id="E289-MJCAL-41" d="M576 668Q576 688 606 708T660 728Q676 728 675 712V571Q675 409 688 252Q696 122 720 57Q722 53 723 50T728 46T732 43T737 41T743 39L754 45Q788 61 803 61Q819 61 819 47Q818 43 814 35Q799 15 755 -7T675 -30Q659 -30 648 -25T630 -8T621 11T614 34Q603 77 599 106T594 146T591 160V163H460L329 164L316 145Q241 35 196 -7T119 -50T59 -24T30 43Q30 75 46 100T74 125Q81 125 83 120T88 104T96 84Q118 57 151 57Q189 57 277 182Q432 400 542 625L559 659H567Q574 659 575 660T576 668ZM584 249Q579 333 577 386T575 473T574 520V581L563 560Q497 426 412 290L372 228L370 224H371L383 228L393 232H586L584 249Z"></path><path stroke-width="0" id="E289-MJSZ2-2211" d="M60 948Q63 950 665 950H1267L1325 815Q1384 677 1388 669H1348L1341 683Q1320 724 1285 761Q1235 809 1174 838T1033 881T882 898T699 902H574H543H251L259 891Q722 258 724 252Q725 250 724 246Q721 243 460 -56L196 -356Q196 -357 407 -357Q459 -357 548 -357T676 -358Q812 -358 896 -353T1063 -332T1204 -283T1307 -196Q1328 -170 1348 -124H1388Q1388 -125 1381 -145T1356 -210T1325 -294L1267 -449L666 -450Q64 -450 61 -448Q55 -446 55 -439Q55 -437 57 -433L590 177Q590 178 557 222T452 366T322 544L56 909L55 924Q55 945 60 948Z"></path><path stroke-width="0" id="E289-MJMAIN-2208" d="M84 250Q84 372 166 450T360 539Q361 539 377 539T419 540T469 540H568Q583 532 583 520Q583 511 570 501L466 500Q355 499 329 494Q280 482 242 458T183 409T147 354T129 306T124 272V270H568Q583 262 583 250T568 230H124V228Q124 207 134 177T167 112T231 48T328 7Q355 1 466 0H570Q583 -10 583 -20Q583 -32 568 -40H471Q464 -40 446 -40T417 -41Q262 -41 172 45Q84 127 84 250Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><g transform="translate(38965,0)"><g id="mjx-eqn-15" transform="translate(0,212)"><use xlink:href="#E289-MJMAIN-28"></use><use xlink:href="#E289-MJMAIN-31" x="389" y="0"></use><use xlink:href="#E289-MJMAIN-35" x="889" y="0"></use><use xlink:href="#E289-MJMAIN-29" x="1389" y="0"></use></g></g><g transform="translate(9395,0)"><g transform="translate(-19,0)"><g transform="translate(0,212)"><use xlink:href="#E289-MJMATHI-71" x="0" y="0"></use><use xlink:href="#E289-MJMAIN-28" x="460" y="0"></use><use xlink:href="#E289-MJMATHI-73" x="849" y="0"></use><use xlink:href="#E289-MJMAIN-2C" x="1318" y="0"></use><use xlink:href="#E289-MJMATHI-61" x="1762" y="0"></use><use xlink:href="#E289-MJMAIN-3B" x="2291" y="0"></use><use xlink:href="#E289-MJMAINB-77" x="2736" y="0"></use><use xlink:href="#E289-MJMAIN-29" x="3567" y="0"></use><use xlink:href="#E289-MJMAIN-3D" x="4234" y="0"></use><use xlink:href="#E289-MJMATHI-76" x="5289" y="0"></use><use xlink:href="#E289-MJMAIN-28" x="5774" y="0"></use><use xlink:href="#E289-MJMATHI-73" x="6163" y="0"></use><use xlink:href="#E289-MJMAIN-3B" x="6632" y="0"></use><use xlink:href="#E289-MJMAINB-77" x="7077" y="0"></use><use xlink:href="#E289-MJMAIN-29" x="7908" y="0"></use><use xlink:href="#E289-MJMAIN-2B" x="8519" y="0"></use><use xlink:href="#E289-MJMATHI-61" x="9520" y="0"></use><use xlink:href="#E289-MJMAIN-28" x="10049" y="0"></use><use xlink:href="#E289-MJMATHI-73" x="10438" y="0"></use><use xlink:href="#E289-MJMAIN-2C" x="10907" y="0"></use><use xlink:href="#E289-MJMATHI-61" x="11351" y="0"></use><use xlink:href="#E289-MJMAIN-3B" x="11880" y="0"></use><use xlink:href="#E289-MJMAINB-77" x="12325" y="0"></use><use xlink:href="#E289-MJMAIN-29" x="13156" y="0"></use><use xlink:href="#E289-MJMAIN-2212" x="13767" y="0"></use><g transform="translate(14545,0)"><g transform="translate(342,0)"><rect stroke="none" width="1495" height="60" x="0" y="220"></rect><use xlink:href="#E289-MJMAIN-31" x="497" y="676"></use><g transform="translate(60,-694)"><use xlink:href="#E289-MJMAIN-7C" x="0" y="0"></use><use xlink:href="#E289-MJCAL-41" x="278" y="0"></use><use xlink:href="#E289-MJMAIN-7C" x="1097" y="0"></use></g></g></g><g transform="translate(16669,0)"><use xlink:href="#E289-MJSZ2-2211" x="0" y="0"></use><g transform="translate(9,-1132)"><use transform="scale(0.707)" xlink:href="#E289-MJMATHI-61" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E289-MJMAIN-2208" x="529" y="0"></use><use transform="scale(0.707)" xlink:href="#E289-MJCAL-41" x="1196" y="0"></use></g></g><use xlink:href="#E289-MJMATHI-61" x="18280" y="0"></use><use xlink:href="#E289-MJMAIN-28" x="18809" y="0"></use><use xlink:href="#E289-MJMATHI-73" x="19198" y="0"></use><use xlink:href="#E289-MJMAIN-2C" x="19667" y="0"></use><use xlink:href="#E289-MJMATHI-61" x="20111" y="0"></use><use xlink:href="#E289-MJMAIN-3B" x="20640" y="0"></use><use xlink:href="#E289-MJMAINB-77" x="21085" y="0"></use><use xlink:href="#E289-MJMAIN-29" x="21916" y="0"></use></g></g></g></g></svg></span></div><script type="math/tex; mode=display" id="MathJax-Element-262">q(s,a;\bold w) = v(s;\bold w) + a(s,a;\bold w) - \frac{1}{|\mathcal A|}\sum_{a \in \mathcal A} a(s,a;\bold w)</script></div></div><p><span>使得等效优势函数 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="45.102ex" height="6.376ex" viewBox="0 -1414.1 19419.1 2745.1" role="img" focusable="false" style="vertical-align: -3.091ex;"><defs><path stroke-width="0" id="E354-MJMATHI-61" d="M33 157Q33 258 109 349T280 441Q331 441 370 392Q386 422 416 422Q429 422 439 414T449 394Q449 381 412 234T374 68Q374 43 381 35T402 26Q411 27 422 35Q443 55 463 131Q469 151 473 152Q475 153 483 153H487Q506 153 506 144Q506 138 501 117T481 63T449 13Q436 0 417 -8Q409 -10 393 -10Q359 -10 336 5T306 36L300 51Q299 52 296 50Q294 48 292 46Q233 -10 172 -10Q117 -10 75 30T33 157ZM351 328Q351 334 346 350T323 385T277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q217 26 254 59T298 110Q300 114 325 217T351 328Z"></path><path stroke-width="0" id="E354-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E354-MJMATHI-73" d="M131 289Q131 321 147 354T203 415T300 442Q362 442 390 415T419 355Q419 323 402 308T364 292Q351 292 340 300T328 326Q328 342 337 354T354 372T367 378Q368 378 368 379Q368 382 361 388T336 399T297 405Q249 405 227 379T204 326Q204 301 223 291T278 274T330 259Q396 230 396 163Q396 135 385 107T352 51T289 7T195 -10Q118 -10 86 19T53 87Q53 126 74 143T118 160Q133 160 146 151T160 120Q160 94 142 76T111 58Q109 57 108 57T107 55Q108 52 115 47T146 34T201 27Q237 27 263 38T301 66T318 97T323 122Q323 150 302 164T254 181T195 196T148 231Q131 256 131 289Z"></path><path stroke-width="0" id="E354-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E354-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E354-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E354-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E354-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E354-MJMAIN-2212" d="M84 237T84 250T98 270H679Q694 262 694 250T679 230H98Q84 237 84 250Z"></path><path stroke-width="0" id="E354-MJMAIN-31" d="M213 578L200 573Q186 568 160 563T102 556H83V602H102Q149 604 189 617T245 641T273 663Q275 666 285 666Q294 666 302 660V361L303 61Q310 54 315 52T339 48T401 46H427V0H416Q395 3 257 3Q121 3 100 0H88V46H114Q136 46 152 46T177 47T193 50T201 52T207 57T213 61V578Z"></path><path stroke-width="0" id="E354-MJMAIN-7C" d="M139 -249H137Q125 -249 119 -235V251L120 737Q130 750 139 750Q152 750 159 735V-235Q151 -249 141 -249H139Z"></path><path stroke-width="0" id="E354-MJCAL-41" d="M576 668Q576 688 606 708T660 728Q676 728 675 712V571Q675 409 688 252Q696 122 720 57Q722 53 723 50T728 46T732 43T737 41T743 39L754 45Q788 61 803 61Q819 61 819 47Q818 43 814 35Q799 15 755 -7T675 -30Q659 -30 648 -25T630 -8T621 11T614 34Q603 77 599 106T594 146T591 160V163H460L329 164L316 145Q241 35 196 -7T119 -50T59 -24T30 43Q30 75 46 100T74 125Q81 125 83 120T88 104T96 84Q118 57 151 57Q189 57 277 182Q432 400 542 625L559 659H567Q574 659 575 660T576 668ZM584 249Q579 333 577 386T575 473T574 520V581L563 560Q497 426 412 290L372 228L370 224H371L383 228L393 232H586L584 249Z"></path><path stroke-width="0" id="E354-MJSZ2-2211" d="M60 948Q63 950 665 950H1267L1325 815Q1384 677 1388 669H1348L1341 683Q1320 724 1285 761Q1235 809 1174 838T1033 881T882 898T699 902H574H543H251L259 891Q722 258 724 252Q725 250 724 246Q721 243 460 -56L196 -356Q196 -357 407 -357Q459 -357 548 -357T676 -358Q812 -358 896 -353T1063 -332T1204 -283T1307 -196Q1328 -170 1348 -124H1388Q1388 -125 1381 -145T1356 -210T1325 -294L1267 -449L666 -450Q64 -450 61 -448Q55 -446 55 -439Q55 -437 57 -433L590 177Q590 178 557 222T452 366T322 544L56 909L55 924Q55 945 60 948Z"></path><path stroke-width="0" id="E354-MJMAIN-2208" d="M84 250Q84 372 166 450T360 539Q361 539 377 539T419 540T469 540H568Q583 532 583 520Q583 511 570 501L466 500Q355 499 329 494Q280 482 242 458T183 409T147 354T129 306T124 272V270H568Q583 262 583 250T568 230H124V228Q124 207 134 177T167 112T231 48T328 7Q355 1 466 0H570Q583 -10 583 -20Q583 -32 568 -40H471Q464 -40 446 -40T417 -41Q262 -41 172 45Q84 127 84 250Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E354-MJMATHI-61" x="0" y="0"></use><g transform="translate(529,-150)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(29.368) matrix(1 0 0 -1 0 0)">等</text><g transform="translate(587,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(29.368) matrix(1 0 0 -1 0 0)">效</text></g></g><use xlink:href="#E354-MJMAIN-28" x="1803" y="0"></use><use xlink:href="#E354-MJMATHI-73" x="2192" y="0"></use><use xlink:href="#E354-MJMAIN-2C" x="2661" y="0"></use><use xlink:href="#E354-MJMATHI-61" x="3106" y="0"></use><use xlink:href="#E354-MJMAIN-3B" x="3635" y="0"></use><use xlink:href="#E354-MJMAINB-77" x="4080" y="0"></use><use xlink:href="#E354-MJMAIN-29" x="4911" y="0"></use><use xlink:href="#E354-MJMAIN-3D" x="5577" y="0"></use><use xlink:href="#E354-MJMATHI-61" x="6633" y="0"></use><use xlink:href="#E354-MJMAIN-28" x="7162" y="0"></use><use xlink:href="#E354-MJMATHI-73" x="7551" y="0"></use><use xlink:href="#E354-MJMAIN-2C" x="8020" y="0"></use><use xlink:href="#E354-MJMATHI-61" x="8465" y="0"></use><use xlink:href="#E354-MJMAIN-3B" x="8994" y="0"></use><use xlink:href="#E354-MJMAINB-77" x="9438" y="0"></use><use xlink:href="#E354-MJMAIN-29" x="10269" y="0"></use><use xlink:href="#E354-MJMAIN-2212" x="10881" y="0"></use><g transform="translate(11659,0)"><g transform="translate(342,0)"><rect stroke="none" width="1495" height="60" x="0" y="220"></rect><use xlink:href="#E354-MJMAIN-31" x="497" y="676"></use><g transform="translate(60,-694)"><use xlink:href="#E354-MJMAIN-7C" x="0" y="0"></use><use xlink:href="#E354-MJCAL-41" x="278" y="0"></use><use xlink:href="#E354-MJMAIN-7C" x="1097" y="0"></use></g></g></g><g transform="translate(13783,0)"><use xlink:href="#E354-MJSZ2-2211" x="0" y="0"></use><g transform="translate(9,-1132)"><use transform="scale(0.707)" xlink:href="#E354-MJMATHI-61" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E354-MJMAIN-2208" x="529" y="0"></use><use transform="scale(0.707)" xlink:href="#E354-MJCAL-41" x="1196" y="0"></use></g></g><use xlink:href="#E354-MJMATHI-61" x="15393" y="0"></use><use xlink:href="#E354-MJMAIN-28" x="15922" y="0"></use><use xlink:href="#E354-MJMATHI-73" x="16311" y="0"></use><use xlink:href="#E354-MJMAIN-2C" x="16780" y="0"></use><use xlink:href="#E354-MJMATHI-61" x="17225" y="0"></use><use xlink:href="#E354-MJMAIN-3B" x="17754" y="0"></use><use xlink:href="#E354-MJMAINB-77" x="18199" y="0"></use><use xlink:href="#E354-MJMAIN-29" x="19030" y="0"></use></g></svg></span><script type="math/tex">\displaystyle a_{等效}(s,a;\bold w) = a(s,a;\bold w) - \frac{1}{|\mathcal A|}\sum_{a\in\mathcal A}a(s,a;\bold w)</script><span> 满足 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="27.407ex" height="5.411ex" viewBox="0 -998.8 11800.3 2329.8" role="img" focusable="false" style="vertical-align: -3.091ex;"><defs><path stroke-width="0" id="E355-MJSZ2-2211" d="M60 948Q63 950 665 950H1267L1325 815Q1384 677 1388 669H1348L1341 683Q1320 724 1285 761Q1235 809 1174 838T1033 881T882 898T699 902H574H543H251L259 891Q722 258 724 252Q725 250 724 246Q721 243 460 -56L196 -356Q196 -357 407 -357Q459 -357 548 -357T676 -358Q812 -358 896 -353T1063 -332T1204 -283T1307 -196Q1328 -170 1348 -124H1388Q1388 -125 1381 -145T1356 -210T1325 -294L1267 -449L666 -450Q64 -450 61 -448Q55 -446 55 -439Q55 -437 57 -433L590 177Q590 178 557 222T452 366T322 544L56 909L55 924Q55 945 60 948Z"></path><path stroke-width="0" id="E355-MJMATHI-61" d="M33 157Q33 258 109 349T280 441Q331 441 370 392Q386 422 416 422Q429 422 439 414T449 394Q449 381 412 234T374 68Q374 43 381 35T402 26Q411 27 422 35Q443 55 463 131Q469 151 473 152Q475 153 483 153H487Q506 153 506 144Q506 138 501 117T481 63T449 13Q436 0 417 -8Q409 -10 393 -10Q359 -10 336 5T306 36L300 51Q299 52 296 50Q294 48 292 46Q233 -10 172 -10Q117 -10 75 30T33 157ZM351 328Q351 334 346 350T323 385T277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q217 26 254 59T298 110Q300 114 325 217T351 328Z"></path><path stroke-width="0" id="E355-MJMAIN-2208" d="M84 250Q84 372 166 450T360 539Q361 539 377 539T419 540T469 540H568Q583 532 583 520Q583 511 570 501L466 500Q355 499 329 494Q280 482 242 458T183 409T147 354T129 306T124 272V270H568Q583 262 583 250T568 230H124V228Q124 207 134 177T167 112T231 48T328 7Q355 1 466 0H570Q583 -10 583 -20Q583 -32 568 -40H471Q464 -40 446 -40T417 -41Q262 -41 172 45Q84 127 84 250Z"></path><path stroke-width="0" id="E355-MJCAL-41" d="M576 668Q576 688 606 708T660 728Q676 728 675 712V571Q675 409 688 252Q696 122 720 57Q722 53 723 50T728 46T732 43T737 41T743 39L754 45Q788 61 803 61Q819 61 819 47Q818 43 814 35Q799 15 755 -7T675 -30Q659 -30 648 -25T630 -8T621 11T614 34Q603 77 599 106T594 146T591 160V163H460L329 164L316 145Q241 35 196 -7T119 -50T59 -24T30 43Q30 75 46 100T74 125Q81 125 83 120T88 104T96 84Q118 57 151 57Q189 57 277 182Q432 400 542 625L559 659H567Q574 659 575 660T576 668ZM584 249Q579 333 577 386T575 473T574 520V581L563 560Q497 426 412 290L372 228L370 224H371L383 228L393 232H586L584 249Z"></path><path stroke-width="0" id="E355-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E355-MJMATHI-73" d="M131 289Q131 321 147 354T203 415T300 442Q362 442 390 415T419 355Q419 323 402 308T364 292Q351 292 340 300T328 326Q328 342 337 354T354 372T367 378Q368 378 368 379Q368 382 361 388T336 399T297 405Q249 405 227 379T204 326Q204 301 223 291T278 274T330 259Q396 230 396 163Q396 135 385 107T352 51T289 7T195 -10Q118 -10 86 19T53 87Q53 126 74 143T118 160Q133 160 146 151T160 120Q160 94 142 76T111 58Q109 57 108 57T107 55Q108 52 115 47T146 34T201 27Q237 27 263 38T301 66T318 97T323 122Q323 150 302 164T254 181T195 196T148 231Q131 256 131 289Z"></path><path stroke-width="0" id="E355-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E355-MJMAIN-3B" d="M78 370Q78 394 95 412T138 430Q162 430 180 414T199 371Q199 346 182 328T139 310T96 327T78 370ZM78 60Q78 85 94 103T137 121Q202 121 202 8Q202 -44 183 -94T144 -169T118 -194Q115 -194 106 -186T95 -174Q94 -171 107 -155T137 -107T160 -38Q161 -32 162 -22T165 -4T165 4Q165 5 161 4T142 0Q110 0 94 18T78 60Z"></path><path stroke-width="0" id="E355-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path><path stroke-width="0" id="E355-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E355-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E355-MJMAIN-30" d="M96 585Q152 666 249 666Q297 666 345 640T423 548Q460 465 460 320Q460 165 417 83Q397 41 362 16T301 -15T250 -22Q224 -22 198 -16T137 16T82 83Q39 165 39 320Q39 494 96 585ZM321 597Q291 629 250 629Q208 629 178 597Q153 571 145 525T137 333Q137 175 145 125T181 46Q209 16 250 16Q290 16 318 46Q347 76 354 130T362 333Q362 478 354 524T321 597Z"></path><path stroke-width="0" id="E355-MJCAL-53" d="M554 512Q536 512 536 522Q536 525 539 539T542 564Q542 588 528 604Q515 616 482 625T410 635Q374 635 349 624T312 594T295 561T290 532Q290 505 303 482T342 442T378 419T409 404Q435 391 451 383T494 357T535 323T562 282T574 231Q574 133 464 56T220 -22Q138 -22 78 21T18 123Q18 184 61 227T156 274Q178 274 178 263Q178 260 177 258Q172 247 164 239T151 227T136 218L127 213L124 202Q118 186 118 163Q120 124 165 86T292 48Q374 48 423 86T473 186V193Q473 267 347 327Q268 364 239 389Q191 431 191 486Q191 547 242 600T356 679T470 705Q472 705 478 705T489 704Q551 704 596 682T642 610Q642 566 621 545Q592 516 554 512Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E355-MJSZ2-2211" x="0" y="0"></use><g transform="translate(9,-1132)"><use transform="scale(0.707)" xlink:href="#E355-MJMATHI-61" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E355-MJMAIN-2208" x="529" y="0"></use><use transform="scale(0.707)" xlink:href="#E355-MJCAL-41" x="1196" y="0"></use></g><g transform="translate(1610,0)"><use xlink:href="#E355-MJMATHI-61" x="0" y="0"></use><g transform="translate(529,-150)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(29.368) matrix(1 0 0 -1 0 0)">等</text><g transform="translate(587,0)"><text font-family="STIXGeneral, 'PingFang SC', serif" stroke="none" transform="scale(29.368) matrix(1 0 0 -1 0 0)">效</text></g></g></g><use xlink:href="#E355-MJMAIN-28" x="3414" y="0"></use><use xlink:href="#E355-MJMATHI-73" x="3803" y="0"></use><use xlink:href="#E355-MJMAIN-2C" x="4272" y="0"></use><use xlink:href="#E355-MJMATHI-61" x="4717" y="0"></use><use xlink:href="#E355-MJMAIN-3B" x="5246" y="0"></use><use xlink:href="#E355-MJMAINB-77" x="5690" y="0"></use><use xlink:href="#E355-MJMAIN-29" x="6521" y="0"></use><use xlink:href="#E355-MJMAIN-3D" x="7188" y="0"></use><use xlink:href="#E355-MJMAIN-30" x="8244" y="0"></use><use xlink:href="#E355-MJMAIN-2C" x="8744" y="0"></use><use xlink:href="#E355-MJMATHI-73" x="9466" y="0"></use><use xlink:href="#E355-MJMAIN-2208" x="10213" y="0"></use><use xlink:href="#E355-MJCAL-53" x="11158" y="0"></use></g></svg></span><script type="math/tex">\displaystyle \sum_{a \in \mathcal A}a_{等效}(s,a;\bold w)=0,\;s \in \mathcal S</script><span> 。</span></p></li></ul><p><em><span>（对偶深度 Q 网络这部分书中描述太少，没怎么看明白，可能不怎么重要，后续有空再查资料补充。）</span></em></p><h3><a name="五案例小车上山mountaincar-v0）" class="md-header-anchor"></a><span>五、案例：小车上山（MountainCar-v0）</span></h3><p><span>本节使用一个经典控制问题：小车上山（MountainCar-v0），gym 库中该环境的相关属性设置可查看其</span><a href='https://github.com/openai/gym/blob/master/gym/envs/classic_control/mountain_car.py' target='_blank' title='MountainCar-v0'><span>源代码</span></a><span>。该问题的控制目标是让小车以尽肯能少的步骤在连续 100 个回合中的平均步数小于等于 110 步，就认为问题解决了。由于智能体施力的大小有限，在所以绝大多数情况下，智能体简单向右施力并不足以让小车成功到达目标位置。</span></p><p><span>该问题的状态空间是连续的，可以将其离散化，然后用形如 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="20.501ex" height="2.903ex" viewBox="0 -915.7 8826.7 1250" role="img" focusable="false" style="vertical-align: -0.776ex;"><defs><path stroke-width="0" id="E357-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E357-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E357-MJMATHI-73" d="M131 289Q131 321 147 354T203 415T300 442Q362 442 390 415T419 355Q419 323 402 308T364 292Q351 292 340 300T328 326Q328 342 337 354T354 372T367 378Q368 378 368 379Q368 382 361 388T336 399T297 405Q249 405 227 379T204 326Q204 301 223 291T278 274T330 259Q396 230 396 163Q396 135 385 107T352 51T289 7T195 -10Q118 -10 86 19T53 87Q53 126 74 143T118 160Q133 160 146 151T160 120Q160 94 142 76T111 58Q109 57 108 57T107 55Q108 52 115 47T146 34T201 27Q237 27 263 38T301 66T318 97T323 122Q323 150 302 164T254 181T195 196T148 231Q131 256 131 289Z"></path><path stroke-width="0" id="E357-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E357-MJMATHI-61" d="M33 157Q33 258 109 349T280 441Q331 441 370 392Q386 422 416 422Q429 422 439 414T449 394Q449 381 412 234T374 68Q374 43 381 35T402 26Q411 27 422 35Q443 55 463 131Q469 151 473 152Q475 153 483 153H487Q506 153 506 144Q506 138 501 117T481 63T449 13Q436 0 417 -8Q409 -10 393 -10Q359 -10 336 5T306 36L300 51Q299 52 296 50Q294 48 292 46Q233 -10 172 -10Q117 -10 75 30T33 157ZM351 328Q351 334 346 350T323 385T277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q217 26 254 59T298 110Q300 114 325 217T351 328Z"></path><path stroke-width="0" id="E357-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E357-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E357-MJMAIN-5B" d="M118 -250V750H255V710H158V-210H255V-250H118Z"></path><path stroke-width="0" id="E357-MJMAINB-78" d="M227 0Q212 3 121 3Q40 3 28 0H21V62H117L245 213L109 382H26V444H34Q49 441 143 441Q247 441 265 444H274V382H246L281 339Q315 297 316 297Q320 297 354 341L389 382H352V444H360Q375 441 466 441Q547 441 559 444H566V382H471L355 246L504 63L545 62H586V0H578Q563 3 469 3Q365 3 347 0H338V62H366Q366 63 326 112T285 163L198 63L217 62H235V0H227Z"></path><path stroke-width="0" id="E357-MJMAIN-5D" d="M22 710V750H159V-250H22V-210H119V710H22Z"></path><path stroke-width="0" id="E357-MJMATHI-54" d="M40 437Q21 437 21 445Q21 450 37 501T71 602L88 651Q93 669 101 677H569H659Q691 677 697 676T704 667Q704 661 687 553T668 444Q668 437 649 437Q640 437 637 437T631 442L629 445Q629 451 635 490T641 551Q641 586 628 604T573 629Q568 630 515 631Q469 631 457 630T439 622Q438 621 368 343T298 60Q298 48 386 46Q418 46 427 45T436 36Q436 31 433 22Q429 4 424 1L422 0Q419 0 415 0Q410 0 363 1T228 2Q99 2 64 0H49Q43 6 43 9T45 27Q49 40 55 46H83H94Q174 46 189 55Q190 56 191 56Q196 59 201 76T241 233Q258 301 269 344Q339 619 339 625Q339 630 310 630H279Q212 630 191 624Q146 614 121 583T67 467Q60 445 57 441T43 437H40Z"></path><path stroke-width="0" id="E357-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E357-MJMATHI-71" x="0" y="0"></use><use xlink:href="#E357-MJMAIN-28" x="460" y="0"></use><use xlink:href="#E357-MJMATHI-73" x="849" y="0"></use><use xlink:href="#E357-MJMAIN-2C" x="1318" y="0"></use><use xlink:href="#E357-MJMATHI-61" x="1762" y="0"></use><use xlink:href="#E357-MJMAIN-29" x="2291" y="0"></use><use xlink:href="#E357-MJMAIN-3D" x="2958" y="0"></use><use xlink:href="#E357-MJMAIN-5B" x="4014" y="0"></use><use xlink:href="#E357-MJMAINB-78" x="4292" y="0"></use><use xlink:href="#E357-MJMAIN-28" x="4899" y="0"></use><use xlink:href="#E357-MJMATHI-73" x="5288" y="0"></use><use xlink:href="#E357-MJMAIN-2C" x="5757" y="0"></use><use xlink:href="#E357-MJMATHI-61" x="6201" y="0"></use><use xlink:href="#E357-MJMAIN-29" x="6730" y="0"></use><g transform="translate(7119,0)"><use xlink:href="#E357-MJMAIN-5D" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E357-MJMATHI-54" x="393" y="513"></use></g><use xlink:href="#E357-MJMAINB-77" x="7995" y="0"></use></g></svg></span><script type="math/tex">q(s,a)=[\bold x(s,a)]^T\bold w</script><span> 的线性组合来近似动作价值函数，求解最优策略。要从连续空间中导出数目有限的特征最简单的方法是采用</span><strong><span>独热编码</span></strong><span>（one-hot coding），</span><strong><span>砖瓦编码</span></strong><span>（tile coding）可以在与独热编码精度相同的情况下减少数目特征，具体内容可查阅相关资料，此处略。</span></p><p><span>代码中的 </span><code>TileCoder</code><span> 实现了砖瓦编码，并将其使用在了智能体类 </span><code>SARSA</code><span> 和 </span><code>SARSALambda</code><span> 中，然后将编码后的特征配合形如 </span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="20.501ex" height="2.903ex" viewBox="0 -915.7 8826.7 1250" role="img" focusable="false" style="vertical-align: -0.776ex;"><defs><path stroke-width="0" id="E357-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E357-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E357-MJMATHI-73" d="M131 289Q131 321 147 354T203 415T300 442Q362 442 390 415T419 355Q419 323 402 308T364 292Q351 292 340 300T328 326Q328 342 337 354T354 372T367 378Q368 378 368 379Q368 382 361 388T336 399T297 405Q249 405 227 379T204 326Q204 301 223 291T278 274T330 259Q396 230 396 163Q396 135 385 107T352 51T289 7T195 -10Q118 -10 86 19T53 87Q53 126 74 143T118 160Q133 160 146 151T160 120Q160 94 142 76T111 58Q109 57 108 57T107 55Q108 52 115 47T146 34T201 27Q237 27 263 38T301 66T318 97T323 122Q323 150 302 164T254 181T195 196T148 231Q131 256 131 289Z"></path><path stroke-width="0" id="E357-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E357-MJMATHI-61" d="M33 157Q33 258 109 349T280 441Q331 441 370 392Q386 422 416 422Q429 422 439 414T449 394Q449 381 412 234T374 68Q374 43 381 35T402 26Q411 27 422 35Q443 55 463 131Q469 151 473 152Q475 153 483 153H487Q506 153 506 144Q506 138 501 117T481 63T449 13Q436 0 417 -8Q409 -10 393 -10Q359 -10 336 5T306 36L300 51Q299 52 296 50Q294 48 292 46Q233 -10 172 -10Q117 -10 75 30T33 157ZM351 328Q351 334 346 350T323 385T277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q217 26 254 59T298 110Q300 114 325 217T351 328Z"></path><path stroke-width="0" id="E357-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E357-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E357-MJMAIN-5B" d="M118 -250V750H255V710H158V-210H255V-250H118Z"></path><path stroke-width="0" id="E357-MJMAINB-78" d="M227 0Q212 3 121 3Q40 3 28 0H21V62H117L245 213L109 382H26V444H34Q49 441 143 441Q247 441 265 444H274V382H246L281 339Q315 297 316 297Q320 297 354 341L389 382H352V444H360Q375 441 466 441Q547 441 559 444H566V382H471L355 246L504 63L545 62H586V0H578Q563 3 469 3Q365 3 347 0H338V62H366Q366 63 326 112T285 163L198 63L217 62H235V0H227Z"></path><path stroke-width="0" id="E357-MJMAIN-5D" d="M22 710V750H159V-250H22V-210H119V710H22Z"></path><path stroke-width="0" id="E357-MJMATHI-54" d="M40 437Q21 437 21 445Q21 450 37 501T71 602L88 651Q93 669 101 677H569H659Q691 677 697 676T704 667Q704 661 687 553T668 444Q668 437 649 437Q640 437 637 437T631 442L629 445Q629 451 635 490T641 551Q641 586 628 604T573 629Q568 630 515 631Q469 631 457 630T439 622Q438 621 368 343T298 60Q298 48 386 46Q418 46 427 45T436 36Q436 31 433 22Q429 4 424 1L422 0Q419 0 415 0Q410 0 363 1T228 2Q99 2 64 0H49Q43 6 43 9T45 27Q49 40 55 46H83H94Q174 46 189 55Q190 56 191 56Q196 59 201 76T241 233Q258 301 269 344Q339 619 339 625Q339 630 310 630H279Q212 630 191 624Q146 614 121 583T67 467Q60 445 57 441T43 437H40Z"></path><path stroke-width="0" id="E357-MJMAINB-77" d="M624 444Q636 441 722 441Q797 441 800 444H805V382H741L593 11Q592 10 590 8T586 4T584 2T581 0T579 -2T575 -3T571 -3T567 -4T561 -4T553 -4H542Q525 -4 518 6T490 70Q474 110 463 137L415 257L367 137Q357 111 341 72Q320 17 313 7T289 -4H277Q259 -4 253 -2T238 11L90 382H25V444H32Q47 441 140 441Q243 441 261 444H270V382H222L310 164L382 342L366 382H303V444H310Q322 441 407 441Q508 441 523 444H531V382H506Q481 382 481 380Q482 376 529 259T577 142L674 382H617V444H624Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E357-MJMATHI-71" x="0" y="0"></use><use xlink:href="#E357-MJMAIN-28" x="460" y="0"></use><use xlink:href="#E357-MJMATHI-73" x="849" y="0"></use><use xlink:href="#E357-MJMAIN-2C" x="1318" y="0"></use><use xlink:href="#E357-MJMATHI-61" x="1762" y="0"></use><use xlink:href="#E357-MJMAIN-29" x="2291" y="0"></use><use xlink:href="#E357-MJMAIN-3D" x="2958" y="0"></use><use xlink:href="#E357-MJMAIN-5B" x="4014" y="0"></use><use xlink:href="#E357-MJMAINB-78" x="4292" y="0"></use><use xlink:href="#E357-MJMAIN-28" x="4899" y="0"></use><use xlink:href="#E357-MJMATHI-73" x="5288" y="0"></use><use xlink:href="#E357-MJMAIN-2C" x="5757" y="0"></use><use xlink:href="#E357-MJMATHI-61" x="6201" y="0"></use><use xlink:href="#E357-MJMAIN-29" x="6730" y="0"></use><g transform="translate(7119,0)"><use xlink:href="#E357-MJMAIN-5D" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E357-MJMATHI-54" x="393" y="513"></use></g><use xlink:href="#E357-MJMAINB-77" x="7995" y="0"></use></g></svg></span><script type="math/tex">q(s,a)=[\bold x(s,a)]^T\bold w</script><span> 的线性近似方法进行迭代求解；经验回放类 </span><code>Replayer</code><span> 实现了经验的存储与均匀回放，并将其使用在了智能体类 </span><code>DQN</code><span> 和 </span><code>DoubleDQN</code><span> 中，然后再配合神经网络近似方法进行迭代求解；这章内容的代码与书中基本一致，此处略。另外值得一提的是，SARSA(</span><span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="1.354ex" height="1.939ex" viewBox="0 -749.6 583 834.7" role="img" focusable="false" style="vertical-align: -0.198ex;"><defs><path stroke-width="0" id="E358-MJMATHI-3BB" d="M166 673Q166 685 183 694H202Q292 691 316 644Q322 629 373 486T474 207T524 67Q531 47 537 34T546 15T551 6T555 2T556 -2T550 -11H482Q457 3 450 18T399 152L354 277L340 262Q327 246 293 207T236 141Q211 112 174 69Q123 9 111 -1T83 -12Q47 -12 47 20Q47 37 61 52T199 187Q229 216 266 252T321 306L338 322Q338 323 288 462T234 612Q214 657 183 657Q166 657 166 673Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E358-MJMATHI-3BB" x="0" y="0"></use></g></svg></span><script type="math/tex">\lambda</script><span>) 算法是针对该问题最有效的方法之一。</span></p><p><em><span>（书上源代码的 </span><code>DQN</code><span> 和 </span><code>DoubleDQN</code><span> 智能体类运行时似乎有些问题，开始的回合很难到达终点结束回合，有时候甚至在第一回合就陷入死循环，目前还没有较好的解决方法，后续有空了再去研究。）</span></em></p><p>&nbsp;</p></div>
</body>
</html>